This study has demonstrated that 100% of published non-Cochrane reviews (n = 39) with a protocol in the PROSPERO underwent alterations during the research process, and these alterations involved all 13 compared method-related sections. All changes to only a single review were considered to improve the reporting/methodology quality, and the remaining 97% of systematic reviews (n = 38) contain changes that were considered to reduce the methodology/reporting quality or have an unclear impact on systematic reviews. Only 8% of the reviews analyzed included reviews (n = 3) [22, 43, 54] that provided reasonable explanations for individual changes in the published full text. These results led us to ask whether these alterations were necessary and whether the value of the protocols that are registered in advance in PROSPERO warrants improvement in order to increase transparency.
The categories and impact of change
Changes in the following sections, including search strategy, study selection, risk of bias assessment, data extraction, and data synthesis, primarily impact the transparency, reproducibility, accuracy, and comprehensiveness of the systematic reviews, thereby improving or reducing the methodology/reporting quality of the reviews[5, 6, 18, 19]. For example, certain alterations to the search strategy, such as expanding the retrieval time range and increasing the retrieval databases, may increase the comprehensiveness of the search strategy and thus improve the overall methodology quality of the review. However, when authors reduce their search databases, narrow the time range of retrieval, and remove supplemental information, searches will affect the comprehensiveness of retrieval and thus reduce the methodology quality. Other alterations such as adding the time range for retrieval can ensure that the literature searches are transparent and reproducible, which is important for assessing the strengths and weaknesses of a systematic review and re-running the literature searches when conducting an update review.
Changes to titles primarily impact the indexing and identification of systematic reviews. For example, including the terms “systematic reviews” or “meta-analysis” may improve indexing and identification of reviews. Moreover, the authors are always encouraged to use informative titles that make key information easily accessible to readers. Thus, providing more accuracy by specifying key information (participants, intervention/exposure, comparator, outcome, type of study) about the scope of the reviews would also improve the quality of systematic reviews.
Changes in the following sections including review questions, participants, intervention/exposure, comparator, outcome, and study design consistently altered the scope of the systematic reviews, and generally, the impact on quality of reviews was not easy to detect. However, the retrospective nature of a systematic review will always put it at some risk of bias because of choices or judgments based on already existing knowledge of the evidence base[5, 6, 59, 60]. For example, when authors make post-protocol modifications to review outcomes (that is, addition, removal, or reprioritization) based on significance of the outcome in the completed review, it can introduce bias into the review process, mislead readers and possibly affect patient care[59, 65]. Several studies have suggested that discrepant outcome reporting between protocols and published Cochrane/non-Cochrane reviews was fairly common and increased the likelihood of reporting statistically significant outcomes[61–64]. Moreover, through awareness of all study characteristics, systematic reviewers might be able to drive the results in different ways regarding participants (e.g., by applying age limits or rules when not all patients in a study met the inclusion criteria of the systematic reviews), intervention and comparison (e.g., by applying duration of treatment limits or rules when not all intervention met the inclusion criteria of the systematic reviews), outcome (e.g., including their definition such as ‘‘using validated scales’’) and definition of study designs (e.g., broadening of the types of studies included beyond randomized controlled trials alone to other potentially less-robust forms of comparison).
To avoid analyses revealing apparently compelling, but in reality spurious subgroup differences, from multiple post hoc analyses, subgroup analysis should follow the principle specified in advance. Therefore, changing the subgroup analysis could be a source of bias, especially without sensitivity analyses to test the effects of such changes.
Improving the transparency and its implementation
It is important to note that the impact of some of the changes may be later reflected when assessing the methodological/reporting quality and/or risk of bias of the systematic reviews in the future, whereas this may not be apparent for others. For example, applying limits to databases or languages might be assessed negatively, resulting in a high risk of bias in this particular area. Definitions of participants, intervention, comparison, outcomes, study design, and subgroup analysis will generally be not easy to detect as potential sources of bias or manipulation. We found that in 90% of non-Cochrane systematic reviews with a pre-registered protocol in PROSPERO, at least one category of the changes had an impact on the quality of reviews that was unclear, and while 8% of reviews explained the reasons for these changes, in these cases, the transparency of non-Cochrane systematic reviews was deemed inadequate. Recently, a study compared 80 non-Cochrane systematic reviews with their published protocols and also found that almost all systematic reviews (92.5%) differed from their protocols in at least one of the methods-related “Preferred reporting items for systematic review and meta-analysis protocols” (PRISMA-P) items and their subcategories, while a mere 7% provided an explanation. Other existing empirical studies investigated difference between non-Cochrane systematic reviews and their PROSPERO records and found that discrepancy in outcome reporting between protocols and published reviews was fairly common[61–64]. Therefore, existing research shows that the differences between the published non-Cochrane reviews and their protocols are indeed quite common, though the reasons for the differences are rarely reported.
Ideally, once a protocol is registered and published, the review should be performed in strict accordance with the protocol. This can effectively reduce the possibility of conscious or subconscious manipulation of inclusion criteria or outcome to reach a desired conclusion. However, in some cases, valid reasons may exist for altering protocols while a review is being conducted. For example, legitimate modifications may extend the period of searches to include older or newer studies, broaden eligibility criteria that initially proved too narrow, or add analyses if the primary analyses suggest that such additional efforts are warranted. Registering a protocol is instrumental for developing good reviews, but it should not constitute an enforcement to paralyze any improvement. Authors should, however, describe the modifications and explain their rationale in the published review to reduce bias and strengthen transparency of the production process[5, 6].
The responsibility of authors to follow their protocols or to clearly explain and rationalize any changes should be highlighted further. Systematic review authors should be encouraged to amend their protocols, and record all changes made with explanation in PROSPERO. We also suggest adding a new item to PRISMA and MOOSE in the future in order to provide the author the ability to report the differences between protocols and the systematic reviews with sufficient explanation. Meanwhile, readers or users of systematic reviews would be more easily made aware of the implications of reviews not following a-priori protocols. In the future, more effective measures to improve the transparency of systematic reviews could include provisions where journals request of the author in their guidelines that any changes (and appropriate reasons) made to the original research protocol are provided in an appendix of the published review. Editors or peer reviewers could compare manuscripts for systematic reviews with their protocols and check back confirm with the authors whether changes have not been reported or explained. The transparency of systematic reviews also requires regulation at a higher level, such as the International Medical Journal Editors' Committee (ICJE), to promote the publication of uniform and standardized biomedical science journals.
The strengths and limitations of this study
This is the first study to compare non-Cochrane systematic reviews with protocols in PROSPERO specifically regarding differences in all methodology-related sections, as opposed to only changes in predefined outcome. We summarized all categories of changes between non-Cochrane systematic reviews (no limitation of intervention/observation) and their pre-registered protocol in PROSPERO and evaluated the impact of all these categories of changes on the reporting/methodology quality of systematic reviews using reliable tools (PRISMA/MOOSE and AMSTRA-2). Furthermore, we quantified the number and impact of all changes that occurred in each non-Cochrane systematic review. Our results therefore represent a precise estimate of the prevalence of differences in the methods between non-Cochrane systematic reviews and their protocols.
Certain limitations of our study must be acknowledged. First, we only included the systematic reviews registered the protocol on PROSPERO within one year and published the full text, it is therefore impossible to observe whether the difference between the protocol and the full text has changed with the promotion of PRISMA-Protocol. Second, due to limited resources, we did not verify these changes and their reasons with each individual author. We only based on the reporting of the systematic reviews and their protocols, and therefore, the judgment of changes was also affected by the quality of the reporting, leaving the possibility that the changes did not reflect “the truth” as experienced by the authors themselves. Third, The generalizability of our results was only limited to the differences between the non-Cochrane systematic reviews and the latest version of protocols registered on PROSPERO. Partial non-Cochrane review protocols recorded on PROSPERO would also be published in peer-reviewed journals, though we did not consider these published protocols.
Implications for future research
Future research should focus on the relationship between the reporting quality of the review protocol registered on PROSPERO and any differences with the actual study. The registry entry itself provides readers with a reference to compare against complete reviews, in the absence of an available protocol, to examine for reporting biases. Thus, the quality of the reporting will affect the judgment of the differences between the protocol and the full text, and therefore affect the transparency of the systematic reviews. Currently, the reporting quality of PROSPERO's protocol may be poor without peer review and quality assessment, and our study supports this hypothesis. In addition, it is necessary to verify these changes as well as the explanations for these changes with the author in order to have a more comprehensive understanding behind the reasons for the changes and their impacts on the quality of the systematic reviews. It is also interesting to observe whether there are differences between the latest version of the protocol published in the peer-reviewed journal and recorded in PROSPERO, and whether there are differences between the two versions and the full text of the systematic reviews.