This study found that 100% of published non-Cochrane reviews (n=39) registered on PROSPERO underwent some changes against their protocols during the research process, pertaining to a total of 50 change categories. All changes to only a single review were considered to improve the reporting/methodology quality, and the remaining 97% of systematic reviews (n=38) contained changes that were considered to reduce the methodology/reporting quality or have an unclear impact on systematic reviews. Only 8% of included reviews (n=3) [22,43,54] provided explanations for individual changes in published full text. These results led us to ask whether the changes were necessary and how to further improve the transparency of non-Cochrane reviews.
The categories and impact of changes
Changes in the following sections, including search strategy, study selection, risk of bias assessment, data extraction, and data synthesis, primarily impacted the transparency, reproducibility, accuracy, and comprehensiveness of systematic reviews, thereby improving or reducing the methodology/reporting quality [5,6,18,19]. In the present study, changes in the “search strategy” section were the most significant (92%, n=36). Certain changes, such as adding manual search of grey literature or increasing search databases, may increase the comprehensiveness of search and thus improve the overall methodology quality of the review. However, when authors deleted manual search of grey literature or reduced search databases, the comprehensiveness of search could be affected and thus reduced the methodology quality. Other changes such as adding the time range for search could ensure that the literature searches were transparent and reproducible, which was important for assessing the strengths and weaknesses of a systematic review and re-running the literature searches when conducting an update review. Changes to titles primarily impacted the indexing and identification of systematic reviews[5]. For example, including the terms “systematic reviews” or “meta-analysis” in titles may improve indexing and identification of reviews. Moreover, the authors were always encouraged to use informative titles that made key information easily accessible to readers[5]. Thus, providing more key information (participants, intervention/exposure, comparator, outcome) about the scope of the reviews would improve the reporting quality of systematic reviews.
Changes in the following sections including review questions, participants, intervention/exposure, comparator, outcome, and study design consistently altered the scope of the systematic reviews and clinical applicability of the results, and generally, the impact on systematic reviews was not easy to detect. However, the retrospective nature of a systematic review would always put it at some risk of bias because of choices or judgments based on already existing knowledge of the evidence base[5,6,59]. 64% of reviews (n=25) exhibited changes in the “outcome” section. When authors make post-protocol modifications to review outcomes (that is, addition, removal, or re-prioritization) based on significance of outcome in the completed review, it can introduce bias into the review process, mislead readers and possibly affect patient care[60-65]. Moreover, through awareness of all study characteristics, systematic reviewers might be able to drive the results in different ways regarding participants (e.g., by applying age limits or rules when not all patients in a study met the inclusion criteria), intervention and comparison (e.g., omission of comparators which show an intervention less effective than the retained comparators), and definition of study designs (e.g., broadening of the types of studies included beyond randomized controlled trials alone to other potentially less-robust forms of comparison). For subgroup analysis, to avoid analyses revealing apparently compelling but in reality spurious subgroup differences, it should follow the principle specified in advance[66]. However, we found that 64% of reviews (n=25) exhibited changes in the “subgroup analysis” section. These changes could be a source of bias, especially without sensitivity analyses to test the effects of such changes.
Improving the transparency and its implementation
It is important to note that the impact of some of the changes may be later reflected when assessing the methodology quality/reporting quality/risk of bias of the systematic reviews in the future, whereas this may not be apparent for others[59]. For example, applying limits to databases or languages might be assessed negatively, resulting in a high risk of publication bias. However, modifying participants, intervention, comparison, outcomes, and subgroup analysis will generally be not easy to detect as potential sources of bias or manipulation. In the present study, we found that 85% of non-Cochrane systematic reviews (n=33) contained at least one category of change whose impact was unclear, and while 8% of reviews explained the reasons for individual changes, in these cases, the transparency of non-Cochrane systematic reviews was deemed inadequate. Recently, a study compared 80 non-Cochrane systematic reviews with their published protocols and also found that almost all systematic reviews (92.5%) differed from their protocols in at least one of the methodology-related “Preferred reporting items for systematic review and meta-analysis protocols” (PRISMA-P) items and their subcategories, while a mere 7% provided an explanation[67]. Additionally, an existing study investigated difference between non-Cochrane systematic reviews and their PROSPERO records and found that a third of systematic reviews changed or did not specify the primary outcome [63]. The existing researches showed that the differences between the published non-Cochrane reviews and their protocols were quite common. However, the explanations for the differences were rarely reported. Furthermore, a previous study [15] compared 47 completed Cochrane reviews with their published protocols using the “document compare” function in Microsoft Word and demonstrated that 91.5% of the reviews (n=43) contained major changes in methodology, and some of changes could be prone to influence by prior knowledge of results. Meanwhile, at least 20% of Cochrane reviews have been found to make post-protocol modifications to review outcomes [59,64]. Cochrane reviews have since evolved to provide a dedicated section in which authors should report any changes made from the documented protocol.
Ideally, once a protocol is registered and published, the review should be performed in strict accordance with the protocol. This can effectively reduce the possibility of conscious or subconscious manipulation of inclusion criteria or outcome to reach a desired conclusion[7]. However, in some cases, valid reasons may exist for altering protocols while a review is being conducted. For example, legitimate modifications may extend the period of searches to include older or newer studies, broaden eligibility criteria that initially proved too narrow, or add analyses if the primary analyses suggest that such additional efforts are warranted. Registering a protocol is instrumental for developing good reviews, but it should not constitute an enforcement to paralyze any improvement. Authors should, however, describe the modifications and explain their rationale in the published review to strengthen the transparency of the review process[5,6].
The following solutions could be considered to strengthen the transparency of non-Cochrane reviews. First, we suggest PROSPERO takes some measures to encourage registrants to amend protocols (recording changes with explanations on PROSPERO). Second, we suggest that the PRISMA and MOOSE working group add a new item in an updated reporting checklist to guide the reviewers to report and explain the differences between protocols and systematic reviews. Meanwhile, readers or users of systematic reviews would be more easily made aware of the implications of reviews not following a-priori protocols by this measure. Third, the more effective measure to improve the transparency of systematic reviews could be regulations at the level of journals. We suggest that journals introduce a requirement in their author guidelines that any changes (with explanations) made to the protocols should be reported in a supplemental file of the published reviews. Editors or peer reviewers could compare manuscripts of systematic reviews with protocols and check back to confirm.
The strengths and limitations of this study
This is the first study to compare non-Cochrane systematic reviews with their protocols on PROSPERO specifically regarding differences in all methodology-related sections, as opposed to only changes in predefined outcome. We identified the changes in all compared methodology-related sections and evaluated their impact on reporting/methodology quality of systematic reviews using reliable tools (PRISMA/MOOSE and AMSTAR2). Furthermore, we created the change categories and quantified the characteristic of impact for all changes in each included review and each compared methodology-related section. Our results therefore represent a precise analysis of the differences in the methods between non-Cochrane systematic reviews and their protocols.
Certain limitations of our study must be acknowledged. First, we identified published non-Cochrane systematic reviews through the final publication details in PROSPERO records. It is therefore possible that the reviews could have been missed if the authors did not update this information. Second, due to limited resources, we did not verify these changes and their reasons with authors. Our comparison only based on the reporting of the systematic reviews and their protocols, therefore leaving the possibility that the changes may not reflect “the truth” as experienced by the authors. For example, PROSPERO’s word limits on each of the record entries may explain why more details were submitted in published full text rather than in protocols. Additionally, Journals limits and specifications on reporting may explain why authors changed their protocols in some cases. Third, the generalizability of our results was only limited to the differences between the non-Cochrane systematic reviews and the latest version of protocols registered on PROSPERO. Partial non-Cochrane review protocols recorded on PROSPERO would also be published in peer-reviewed journals, but we did not identify whether a published one existed and whether the changes were highlighted in it.
Implications for future research
Future research should focus on the reporting quality of the review protocols registered on PROSPERO. The registry entry itself provides readers with a reference to compare against complete reviews, to examine for reporting biases[59]. Thus, the reporting quality will affect the judgment of the differences between the protocol and the full text, and therefore affect the transparency of the systematic reviews. Based on our results, we have some concern over the reporting quality of PROSPERO's protocol. During the process of comparison, we found that many protocols consisted of limited key information, such as search strategy, participants, interventions, comparators, and outcomes. Alarmingly, key information was often absent, even for mandatory fields such as study selection, data extraction, risk of bias assessment, and strategy for data synthesis. These absence may explain the vast improvement of included systematic reviews. However, the reporting issue of protocols could rarely be explained by PROSPERO’s word limits on each of the record entries. Furthermore, key information error was found in one review protocol (confusing the exposure and outcome[24]). We believe it’s also necessary to verify these changes, as well as investigate the reasons through correspondence with the authors. Moreover, it is interesting to investigate how many PROSRERO records have been published in peer-reviewed journals, and whether two versions of protocol exist differences.