Inconsistencies are common between non-Cochrane systematic reviews and their protocols registered in the International prospective register of systematic reviews (PROSPERO) but were seldom explained

Background: Protocols of systematic reviews allow for and thus improve the transparency of the reviews process. However, pre-registration of a protocol is not enough, the author also need to follow it. PROSPERO is an open-access online database for the registration of non-Cochrane systematic reviews. The purpose of this study is to compare published non-Cochrane reviews with their pre-registered protocols on PROSPERO to determine what changes, if any, have been made, and how likely these changes are to impact the quality of systematic review. Methods: This is a retrospective comparative study. We searched for protocols on PROSPERO platform that were registered in 2018 and then selected the protocols that full text have been published as of January 1st 2019. Published full texts were identified through the protocol's final publication citation. Two authors independently compared and identified changes between protocols and systematic reviews and then evaluated the impact (improve, reduce, or unclear) of these changes on the reporting or methodology quality of reviews. Descriptive statistics of percentage (%) and frequency (n) were conducted. Results: We identified 39 pairs, all of which exhibited changes. “Search strategy”(92%, n=36), “data extraction”(90%, n=35), “data synthesis”(77%, n=30), “outcome”(64%, n=24), and “subgroup analysis”(64%, n=24) all showed significant changes. All changes to only one review were considered to improve the reporting or methodology quality, and the remaining 97% of reviews (n=38) contained changes that were considered to reduce the methodology or reporting quality or that had an unclear impact on systematic reviews. Conclusions: Changes between the non-Cochrane systematic reviews and their protocols recorded on PROSPERO were widespread. Some of the changes reduced the methodology or reporting quality of systematic reviews or had an unclear impact. Measures should be taken to further improve the transparency of the non-Cochrane systematic reviews. Adding a new item in updated “Preferred Reporting Items for Systematic reviews and Meta-Analyses” (PRISMA) and “Meta-analysis of Observational Studies in Epidemiology” (MOOSE) to guide reporting and explaining the changes, as well as advising peer reviewers (and editors) to check the reviews against the protocols are two suggested fundamental solutions.

Systematic review is the cornerstone of evidence-based health care and is widely promoted as providing the best evidence for informed decision-making [1][2][3]. However, due to the nature of retrospective studies, the problem of selective reporting outcomes and selective inclusion must be addressed at the level of systematic reviews [4][5][6]. Protocols of systematic reviews allow for planning and documentation of review methods, serve as guards against arbitrary decision making during review conduct, and enable readers to assess the presence of selective reporting against completed reviews [5][6][7][8].
Prior to 2011, only a few organizations, including the Cochrane Collaboration Organization and the Joanna Briggs Institute (JBI), provided a registration platform for systematic reviews to reduce the occurrence of bias and ensure the transparency of the systematic review production process. Some researchers evaluated the methodological quality of non-Cochrane and Cochrane systematic reviews and found that Cochrane reviews as a whole are of higher methodological quality than non-Cochrane reviews published in peer-reviewed journals [9,10]. However these organizations only publish a minority of systematic reviews [3], and there are still no mandatory registration policies for most systematic reviews. To effectively fill the vacancy of systematic review registration platforms and improve the transparency of non-Cochrane systematic reviews, the international prospective register of systematic reviews (PROSPERO) was developed and launched by the United Kingdom Centre for Reviews and Dissemination in 2011 [11,12]. PROSPERO is an open-access online database for the registration of non-Cochrane systematic reviews. Registration on PROSPERO involves submission and publication of key information about the design and conduct of a review, which includes information on 22 mandatory and 18 optional items [11,13] that were selected according to international consultation [14].
Recent data suggests that more than 8,000 systematic reviews are indexed in MEDLINE annually, representing a three-fold increase over the last decade [3]. Meanwhile, the differences between the published full-text and the pre-registered protocols have become a noteworthy issue. In 2002, an examination of Cochrane reviews revealed indirect evidence for possible selective reporting bias in systematic reviews [15]. The study compared 47 completed Cochrane reviews with their published protocols and demonstrated that 91.5% of the reviews (n=43) contained major changes in methodology. Up to December 31 st , 2018, there were a total of 45,638 systematic reviews registered on PROSPERO, among which 3,976 were published [16]. To our knowledge, no study to date has compared the texts of published systematic reviews with their protocols registered on PROSPERO.
Additionally, unlike the Cochrane library's rigorous assessment of registered protocols, PROSPERO has no quality assessment or peer review process for submitted protocols [17]. As such, the purpose of our study is to compare published non-Cochrane reviews with their pre-registered protocols on PROSPERO to determine what changes, if any, have been made to methodology-related sections and how likely these changes may impact the quality of systematic review.

Eligibility criteria
We included systematic reviews that met the following criteria: (1) pre-registered on PROSPERO between January 1 st and December 31 st of 2018; (2) included intervention studies (randomized or non-randomized controlled trials) or observational studies (e.g., cohort studies, case-control studies, and cross-sectional studies); (3) were published in English. We excluded Cochrane systematic reviews, JBI systematic reviews, and systematic reviews which included non-clinical studies.

Identification and selection of protocols and reviews
The search strategy was developed and executed by one author (K-YH) on January 1 st 2019. The following filters of PROSPERO platform were used to initially identify eligible protocols: (1) Source of the review, all protocols excluded Cochrane protocols; (2) Status of review, published; (3) Data added to PROSPERO, 01/01/2018-31/12/2018. These records most often included a citation and link to the final publication. For invalid link, we manually searched for the reviews in open databases (PubMed, Embase, and Web of Science) using the title of final publication. After identification of a published review, we obtained the latest version of protocol. Then, two authors (K-YH and W-YZ) examined the full-text of all "pairs" according to the eligibility criteria. If the author found two protocols registered for one published review, the corresponding author was contacted to confirm which protocol should be considered. Any remaining doubts were resolved through discussion with the third author (BM).

Difference assessment
For all included pairs, two authors (K-YH and TZ) independently examined relevant text, including related supplementary files, to compare methodology-related sections and to assess whether a difference existed. The difference assessment was based on an internal guideline that was developed, independently pilot-tested (in n=10 pairs), and revised by two authors (K-YH and BM) (Additional file 1). Before the formal assessment, a random sample of ten pairs was assessed by the two authors (K-YH and TZ), and the assessment did not commence until high agreement (>90%) was achieved. In case where differences were observed, two authors (K-YH and TZ) independently examined if the differences had been reported and explained in published reviews or the latest protocols. Any remaining disagreement was resolved through discussion with the third author (BM).
The following 13 methodology-related sections were compared and assessed: title, review question, search strategy, participants/population, intervention(s)/exposure(s), comparator(s)/control, types of study design, primary and additional outcome(s), study selection, data extraction, risk of bias (quality) assessment, strategy for data synthesis, and analysis of subgroups/subsets. These 13 sections are the mandatory registration entries required by PROSPERO.

Data extraction
Data extraction forms were developed in Microsoft Office Word 2016 software. The data were extracted independently and cross-checked by two authors (K-YH and TZ). For each pair, the text details where a difference existed were extracted from both protocol and publication. If the explanation for the difference was available, it was also extracted. Any discrepancies were discussed and resolved with the third author (BM).

Identification of change's impact and categories
The extracted text describing the difference was initially defined by one author (K-YH) to one of following types of change: adding, deleting, or modifying. Then, two authors (K-YH and QZ) independently evaluated the impact of these types of change on quality of systematic reviews. The impact on quality of systematic reviews included: (1) improving reporting or methodology quality; (2) reducing reporting or methodology quality; (3) unclear (the impact of changes on quality of systematic reviews can't be determined). Table 1 contains detailed information on this evaluation. Any changes where an explanation was given, were considered to improve the reporting quality. We did not evaluate the contents of any changes (literature management software, statistical software, method to grade the quality of evidence, etc.) which was not mentioned in the current version of PRISMA [5] or MOOSE [18]or "A critical appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both" (AMSTAR2) [19]. The two authors (K-YH and QZ) cross-checked the evaluation results and discussed the discrepancies with the third author (BM). Based on the comparison and evaluation results , two authors (K-YH and QZ) summarized the final change categories for each methodology related section by discussion with the third author (BM). Improving the transparence, e.g., adding the information sources in the search (such as databases with dates of coverage, contact with study authors to identify additional studies) and date last searched Improving indexing and identification, e.g., inclusion of the terms "systematic review" or/and "meta-analysis" in the title Adding the required method process of AMSTAR2 Improve methodology quality Reducing the risk of bias, e.g., adding the manual search of grey literature; adding consensus process when disagreements arose in study selection Improving the determination of the applicability of the results, e.g., adding the detailed eligibility criteria of PICO Deleting the minimal list items of PRISMA/MOOSE or/and additional desirable information

Reduce reporting quality
Reducing the transparence, e.g., deleting the information sources in the search (such as databases with dates of coverage, contact with study authors to identify additional studies) and date last searched Reducing indexing and identification, e.g., exclusion of the terms "systematic review" or/and "meta-analysis" in the title Deleting the required method process of AMSTAR2

Reduce methodology quality
Increasing the risk of bias, e.g., deleting the manual search of grey literature; deleting consensus process when disagreements arose in study selection Impacting the determination of the applicability of the results, e.g., deleting the detailed eligibility criteria of PICO

Improve methodology quality
Reducing the risk of bias: e.g., increasing databases, expanding time range of search, adding assessors performing data extraction

Reduce methodology quality
Increasing the risk of bias: e.g., reducing databases, narrowing time range of search, reducing assessors performing data extraction

Unclear
The impact on quality of systematic reviews can't be determined, e.g., modifying participants, interventions, comparators, outcomes, types of study design included, subgroup analysis; adding/deleting individual outcomes; deleting the distinction between primary and secondary outcome PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses; MOOSE: Meta-analysis of Observational Studies in Epidemiology; AMSTAR2: A critical appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both; PICO:

Data analysis
We performed a descriptive analysis using Microsoft Office Excel 2016 software. Results were reported as percentage (%) and frequency (n).

Search results
15,667 non-Cochrane protocol records were registered on PROSPERO from January 1 st to December 31 st of 2018, and 43 reviews were completed and published on 1 st January 2019. Of the 43 potentially relevant reviews, three reviews were excluded, two of which included non-clinical studies and one that was post-registered. We found one review that registered two different protocols (with the registration numbers CRD42018086539 and CRD42018084876), and determined the relevant one by contacting the author (Additional file 2). All excluded reviews are listed in Additional file 3. 39 systematic reviews  fulfilled the eligibility criteria and were subsequently included.

Categories and impact of changes in each section
All changes in the systematic reviews compared to their protocols were compiled (see Additional file 4). These changes involved all compared methodology-related sections, and the change categories as well as references reporting them were shown in Table 2. The "search strategy" section exhibited the greatest variation, with changes in 92% of the reviews (n=36). The "data extraction" and "data synthesis" sections also exhibited significant changes, with 90% of reviews (n=35) exhibiting changes in data extraction and 77% of reviews (n=30) exhibiting changes in data synthesis. Changes in the above sections primarily improved the methodology or reporting quality of systematic reviews.
Moreover, 64% of reviews (n=25) exhibited changes in the "outcome" section and another 64% of reviews (n=25) exhibited changes in the "subgroup analysis" section. The changes in the above sections most often had an unclear impact on systematic reviews. Characteristic of impact for all changed reviews in each compared section is shown in Fig.1. Only 8% of included reviews (n=3) [22,43,54] provided the explanations for individual changes in the published full text and thus were considered to improve the reporting quality. Inclusion of the terms "systematic reviews" and "Meta-analysis" in review title [33] [43] [49] Exclusion of the terms "systematic reviews" and 'Meta-analysis" from review title [40] [46] The key information (PICO) about the scope of the reviews was provided [21]

Categories and impact of changes in each review
All of the systematic reviews included in this study (n=39) have undergone some changes, pertaining to a total of 50 change categories (in Table 1). The most rigorous systematic review [50] involved four categories of changes, and the most varied systematic reviews [33] involved 14 categories of changes. The changes in 70% systematic reviews (n=27) have improved the quality as a whole (the number of improving being greater than the number of reducing and unclear). All changes to only one review were considered to improve the reporting/methodology quality (adding time range of search, adding list and definition of all variables for which data were sought, adding eligibility criteria of publication status and publication type), and remaining the 97% of systematic reviews (n=38) contained changes that were considered to reduce the methodology/reporting quality or that had an unclear impact. 85% of systematic reviews (n=33) contained at least one change that was considered to reduce the methodology/reporting quality. Furthermore, 85% of systematic reviews (n=33) contained at least one change whose impact was unclear. Characteristic of impact for all changes in each included review is shown in Fig.2.

Discussion
This study found that 100% of published non-Cochrane reviews (n=39) registered on PROSPERO underwent some changes against their protocols during the research process, pertaining to a total of 50 change categories. All changes to only a single review were considered to improve the reporting/methodology quality, and the remaining 97% of systematic reviews (n=38) contained changes that were considered to reduce the methodology/reporting quality or have an unclear impact on systematic reviews. Only 8% of included reviews (n=3) [22,43,54] provided explanations for individual changes in published full text. These results led us to ask whether the changes were necessary and how to further improve the transparency of non-Cochrane reviews.

The categories and impact of changes
Changes in the following sections, including search strategy, study selection, risk of bias assessment, data extraction, and data synthesis, primarily impacted the transparency, reproducibility, accuracy, and comprehensiveness of systematic reviews, thereby improving or reducing the methodology/reporting quality [5,6,18,19]. In the present study, changes in the "search strategy" section were the most significant (92%, n=36). Certain changes, such as adding manual search of grey literature or increasing search databases, may increase the comprehensiveness of search and thus improve the overall methodology quality of the review. However, when authors deleted manual search of grey literature or reduced search databases, the comprehensiveness of search could be affected and thus reduced the methodology quality. Other changes such as adding the time range for search could ensure that the literature searches were transparent and reproducible, which was important for assessing the strengths and weaknesses of a systematic review and re-running the literature searches when conducting an update review. Changes to titles primarily impacted the indexing and identification of systematic reviews [5]. For example, including the terms "systematic reviews" or "meta-analysis" in titles may improve indexing and identification of reviews. Moreover, the authors were always encouraged to use informative titles that made key information easily accessible to readers [5]. Thus, providing more key information (participants, intervention/exposure, comparator, outcome) about the scope of the reviews would improve the reporting quality of systematic reviews.
Changes in the following sections including review questions, participants, intervention/exposure, comparator, outcome, and study design consistently altered the scope of the systematic reviews and clinical applicability of the results, and generally, the impact on systematic reviews was not easy to detect. However, the retrospective nature of a systematic review would always put it at some risk of bias because of choices or judgments based on already existing knowledge of the evidence base [5,6,59]. 64% of reviews (n=25) exhibited changes in the "outcome" section. When authors make post-protocol modifications to review outcomes (that is, addition, removal, or re-prioritization) based on significance of outcome in the completed review, it can introduce bias into the review process, mislead readers and possibly affect patient care [60][61][62][63][64][65]. Moreover, through awareness of all study characteristics, systematic reviewers might be able to drive the results in different ways regarding participants (e.g., by applying age limits or rules when not all patients in a study met the inclusion criteria), intervention and comparison (e.g., omission of comparators which show an intervention less effective than the retained comparators), and definition of study designs (e.g., broadening of the types of studies included beyond randomized controlled trials alone to other potentially less-robust forms of comparison). For subgroup analysis, to avoid analyses revealing apparently compelling but in reality spurious subgroup differences, it should follow the principle specified in advance [66]. However, we found that 64% of reviews (n=25) exhibited changes in the "subgroup analysis" section. These changes could be a source of bias, especially without sensitivity analyses to test the effects of such changes.

Improving the transparency and its implementation
It is important to note that the impact of some of the changes may be later reflected when assessing the methodology quality/reporting quality/risk of bias of the systematic reviews in the future, whereas this may not be apparent for others [59]. For example, applying limits to databases or languages might be assessed negatively, resulting in a high risk of publication bias. However, modifying participants, intervention, comparison, outcomes, and subgroup analysis will generally be not easy to detect as potential sources of bias or manipulation. In the present study, we found that 85% of non-Cochrane systematic reviews (n=33) contained at least one category of change whose impact was unclear, and while 8% of reviews explained the reasons for individual changes, in these cases, the transparency of non-Cochrane systematic reviews was deemed inadequate. Recently, a study compared 80 non-Cochrane systematic reviews with their published protocols and also found that almost all systematic reviews (92.5%) differed from their protocols in at least one of the methodology-related "Preferred reporting items for systematic review and meta-analysis protocols" (PRISMA-P) items and their subcategories, while a mere 7% provided an explanation [67]. Additionally, an existing study investigated difference between non-Cochrane systematic reviews and their PROSPERO records and found that a third of systematic reviews changed or did not specify the primary outcome [63]. The existing researches showed that the differences between the published non-Cochrane reviews and their protocols were quite common. However, the explanations for the differences were rarely reported. Furthermore, a previous study [15] compared 47 completed Cochrane reviews with their published protocols using the "document compare" function in Microsoft Word and demonstrated that 91.5% of the reviews (n=43) contained major changes in methodology, and some of changes could be prone to influence by prior knowledge of results. Meanwhile, at least 20% of Cochrane reviews have been found to make post-protocol modifications to review outcomes [59,64]. Cochrane reviews have since evolved to provide a dedicated section in which authors should report any changes made from the documented protocol.
Ideally, once a protocol is registered and published, the review should be performed in strict accordance with the protocol. This can effectively reduce the possibility of conscious or subconscious manipulation of inclusion criteria or outcome to reach a desired conclusion [7]. However, in some cases, valid reasons may exist for altering protocols while a review is being conducted. For example, legitimate modifications may extend the period of searches to include older or newer studies, broaden eligibility criteria that initially proved too narrow, or add analyses if the primary analyses suggest that such additional efforts are warranted. Registering a protocol is instrumental for developing good reviews, but it should not constitute an enforcement to paralyze any improvement.
Authors should, however, describe the modifications and explain their rationale in the published review to strengthen the transparency of the review process [5,6].
The following solutions could be considered to strengthen the transparency of non-Cochrane reviews.
First, we suggest PROSPERO takes some measures to encourage registrants to amend protocols (recording changes with explanations on PROSPERO). Second, we suggest that the PRISMA and MOOSE working group add a new item in an updated reporting checklist to guide the reviewers to report and explain the differences between protocols and systematic reviews. Meanwhile, readers or users of systematic reviews would be more easily made aware of the implications of reviews not following a-priori protocols by this measure. Third, the more effective measure to improve the transparency of systematic reviews could be regulations at the level of journals. We suggest that journals introduce a requirement in their author guidelines that any changes (with explanations) made to the protocols should be reported in a supplemental file of the published reviews. Editors or peer reviewers could compare manuscripts of systematic reviews with protocols and check back to confirm.

The strengths and limitations of this study
This is the first study to compare non-Cochrane systematic reviews with their protocols on PROSPERO specifically regarding differences in all methodology-related sections, as opposed to only changes in predefined outcome. We identified the changes in all compared methodology-related sections and evaluated their impact on reporting/methodology quality of systematic reviews using reliable tools (PRISMA/MOOSE and AMSTAR2). Furthermore, we created the change categories and quantified the characteristic of impact for all changes in each included review and each compared methodologyrelated section. Our results therefore represent a precise analysis of the differences in the methods between non-Cochrane systematic reviews and their protocols.
Certain limitations of our study must be acknowledged. First, we identified published non-Cochrane systematic reviews through the final publication details in PROSPERO records. It is therefore possible that the reviews could have been missed if the authors did not update this information. Second, due to limited resources, we did not verify these changes and their reasons with authors. Our comparison only based on the reporting of the systematic reviews and their protocols, therefore leaving the possibility that the changes may not reflect "the truth" as experienced by the authors. For example, PROSPERO's word limits on each of the record entries may explain why more details were submitted in published full text rather than in protocols. Additionally, Journals limits and specifications on reporting may explain why authors changed their protocols in some cases. Third, the generalizability of our results was only limited to the differences between the non-Cochrane systematic reviews and the latest version of protocols registered on PROSPERO. Partial non-Cochrane review protocols recorded on PROSPERO would also be published in peer-reviewed journals, but we did not identify whether a published one existed and whether the changes were highlighted in it.

Implications for future research
Future research should focus on the reporting quality of the review protocols registered on PROSPERO. The registry entry itself provides readers with a reference to compare against complete reviews, to examine for reporting biases [59]. Thus, the reporting quality will affect the judgment of the differences between the protocol and the full text, and therefore affect the transparency of the systematic reviews. Based on our results, we have some concern over the reporting quality of PROSPERO's protocol. During the process of comparison, we found that many protocols consisted of limited key information, such as search strategy, participants, interventions, comparators, and outcomes. Alarmingly, key information was often absent, even for mandatory fields such as study selection, data extraction, risk of bias assessment, and strategy for data synthesis. These absence may explain the vast improvement of included systematic reviews. However, the reporting issue of protocols could rarely be explained by PROSPERO's word limits on each of the record entries.
Furthermore, key information error was found in one review protocol (confusing the exposure and outcome [24]). We believe it's also necessary to verify these changes, as well as investigate the reasons through correspondence with the authors. Moreover, it is interesting to investigate how many PROSRERO records have been published in peer-reviewed journals, and whether two versions of protocol exist differences.

Conclusions
Changes between the non-Cochrane systematic reviews and their protocols recorded on PROSPERO were widespread. "Search strategy", "data extraction", "data synthesis", "outcome", and "subgroup analysis" all showed significant changes. Some of the changes reduced the methodology or reporting quality of systematic reviews or had an unclear impact. Measures should be taken to further improve the transparency of the non-Cochrane systematic reviews. Adding new item in updated PRISMA and MOOSE to guide reporting and explaining any changes, as well as advising peer reviewers (and editors) to check the reviews against the protocols are two suggested fundamental solutions.
Abbreviations PROSPERO: International prospective register of systematic reviews; PRISMA: Preferred reporting items for systematic review and meta-analysis; PRISMA-P: Preferred reporting items for systematic review and meta-analysis protocols; MOOSE: Meta-analysis of Observational Studies in Epidemiology; AMSTAR-2: a critical appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both; PICO: participant(s), intervention(s), comparator(s), outcome(s).

Ethics approval and consent to participate
Not applicable.

Consent for publication
Not applicable.

Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Funding
Not applicable.  Figure 1 <p><em>Categories and impact of changes in each review</em></p> <p>All of the systematic reviews included in this study (n=39) have undergone some changes, pertaining to a total of 50 change categories (in Table 1). The most rigorous systematic review [50] involved four categories of changes, and the most varied systematic reviews [33] involved 14 categories of changes. The changes in 70% systematic reviews (n=27) have improved the quality as a whole (the number of improving being greater than the number of reducing and unclear). All changes to only one review were considered to improve the reporting/methodology quality (adding time range of search, adding list and definition of all Characteristic of impact for all changes (x axis) in each included review (y axis)

Figures
Supplementary Files