Impact of industry sponsorship on the quality and interpretation of systematic reviews of vaccines: a cross-sectional analysis of studies published from 2016–2019

Systematic reviews (SRs) provide the highest level of evidence and inform evidence-based decision making in health care. Earlier studies found association with industry to be negatively associated with methodological quality of SRs. However, this has not been investigated in SRs on vaccines. We performed a systematic literature search using MEDLINE and EMBASE in March 2020. The results were restricted to those published between 2016 to 2019 with no language restrictions. Study characteristics were extracted by one person and checked by an experienced reviewer. The methodological quality of the SRs was assessed with the AMSTAR 2 tool by multiple reviewers after a calibration exercise was performed. A summary score for each SR was calculated. The Mann-Whitney U-Test and Fisher’s Exact Test were performed to compare both groups.


Introduction
Systematic reviews (SRs) provide the highest level of evidence and inform evidence-based decision making in health care. The number of SRs in the eld of vaccines increased steadily over the last decade (1). In contrast to other topics, SRs in the eld of vaccination come with several unique methodological challenges. In particular, this includes the consideration of non-randomized studies, differentiation between outcomes such as immunogenicity and e cacy or effectiveness, safety outcomes, and agespeci c effects (2)(3)(4)(5)(6).
According to the World Health Organization (WHO), National Immunization Technical Advisory Groups (NITAGs) should apply evidence-based approaches in the process of developing vaccination recommendations (7,8). Given their central importance in this process, SRs with methodological aws might bias decisions and negatively affect population health. Research has repeatedly shown that the methodological quality of SRs is generally suboptimal, while too many SRs are published on the same topics at the same time (9). This makes the use of evidence syntheses such as SRs much more complicated (1). Methodological quality assessment tools can facilitate choosing the most suitable SR for decision-making, although contextual factors such as PICOS (population, intervention, comparator, outcome, and setting) alignment can also play a role.
The methodological quality of SRs can be associated with several aspects. For example, aspects positively associated with the methodological quality were found to be prospective registration of the SRs (10)(11)(12), performing meta-analysis (11,13), including randomized controlled trials in the SRs (13), SRs that were Cochrane reviews (13), and journal impact factor (14) across different elds. Funding can also play a role: a recent Cochrane review found that SRs on drugs or devices with nancial con icts of interest more often have favorable conclusions and tend to have lower methodological quality than SRs without nancial con icts of interest (15). Earlier studies found association with industry to be negatively associated with methodological quality of SRs (16, 17). However, they did not consider SRs on vaccines.
The aim of this study was to compare SRs of vaccine intervention studies that were sponsored by industry, either through authorship, funding, or a combination thereof, compared to non-industry sources to evaluate whether industry sponsorship in uences the methodological quality of SRs of vaccines.
Understanding the relationship between funding and quality can help NITAGs to choose among available SRs to support the development of vaccination recommendations. This might also be important during the ongoing SARS-CoV-2 pandemic since vaccine studies are quickly multiplying and soon SRs will emerge, including from industry, to inform program policies. Methods This is a cross-sectional study using a matched design. There was an unpublished a priori protocol for this study (see appendix 1).

Eligibility criteria
To meet eligibility criteria, SRs had to search bibliographic databases and identify and select relevant studies based on speci ed eligibility criteria. To be eligible, a SR had to investigate at least one of the following outcomes: (1) e cacy, effectiveness; (2) safety; or (3) immunogenicity of vaccines. All populations and settings were considered. The intervention of interest was vaccination in humans with one or more speci c vaccines or with a class of vaccines (e.g., vaccines against pneumococcal disease).
Therapeutic vaccines and passive immunization agents were not considered. Empty SRs (i.e., SRs having no eligible inclusions) were excluded as well as clinical practice guidelines, health technology assessment reports, and other types of reviews (e.g. narrative, scoping) together with burden of disease studies, economic assessments of vaccination, risk assessments of vaccination, vaccine-related modeling, qualitative assessments of vaccination and vaccine program evaluation even if they performed literature searches. In addition, we only considered SRs published between 2016 and 2019 to focus on current trends and standards.

Information sources and search strategy
We performed a systematic literature search using MEDLINE and EMBASE on the OVID platform. The search strategy was developed in collaboration with a librarian from the Health Library of Health Canada and the Public Health Agency of Canada (EW) and was peer reviewed by a second librarian using the PRESS statement (18). The search combined vaccine and immunization keywords and subject headings with an adaption of the CADTH systematic review lter (19). The results were restricted to those published between 2016 to 2019 with no language restrictions. The search was performed on March 30, 2020. The complete search strategy can be found in appendix 2.

Study selection
References were screened by title and abstract rst. The full text of articles deemed relevant was retrieved and assessed for eligibility. Piloting of the study selection process took place for both title/abstract and full-text screening. At each step screening was performed by one reviewer (CB, GP, IH, SaS) and veri ed by a second reviewer (DP). Differences were discussed until consensus was reached or a third reviewer was brought in to mediate. SRs were screened by year, i.e. each rst reviewer (CB, GP, IH, SaS) examined only one year.
SRs matching was applied, as the number of industry-funded SRs was expected to be smaller than the number of non-industry funded SRs. For each industry-funded SR (case) a control was randomly drawn (using the RAND function in Excel) from non-industry funded SRs published in the same year. We planned to apply 1:1 matching. However, 1:2 matching was applied in the 2016 subset due a low number of cases.

Data extraction
After piloting of data extraction sheets, each reviewer (CB, GP, IH, SaS) abstracted data from eligible studies into an Excel extraction form and data were checked by a second reviewer (DP). Again, differences were discussed until consensus was reached or through third reviewer mediation. We abstracted data for general characteristics, impact factor (2019), included studies, searched databases, quality appraisal tool, and evidence synthesis.

Quality assessment
All SRs were assessed with the AMSTAR 2 tool (20). AMSTAR 2 builds up on the former used AMSTAR (21), and has been designed to assess the methodological quality of SRs. AMSTAR 2 consists of 16 items allowing for the answer "yes" or "no", and "partial yes" for some items. AMSTAR 2 allows for rating the overall con dence in the results and was found to be valid and moderately reliable (22)(23)(24)(25). A calibration exercise among all reviewers (CB, GP, IH, SaS) and a senior author (DP) experienced with SRs and AMSTAR 2 was performed on 4 SRs not meeting eligibility criteria to ensure consistency. In addition, an internal guidance document was developed, approved, and used throughout the process. A single reviewer (CB, GP, IH, SaS) assessed the methodological quality of the included SRs. Where needed, reviewers reached out to the senior author in case of any uncertainty where no guidance was available either from the AMSTAR 2 tool guidance document or the internal guidance document. Those who were assessing the study quality were not blinded to the industry status of each study, due to logistical constraints.

Data analysis
We did not assess the overall con dence using AMSTAR 2 due to potential oor effects (26-28) and to allow for item-level analyses but did rely on summary scores. We calculated a summary score for each SR. Items from the AMSTAR 2 tool scoring "yes" obtained 1 point, whereas items scoring "partial yes" obtained 0.5 points. Due to the fact that the number of relevant items can vary (e.g. items on metaanalysis can only be judged if meta-analysis was performed) we divided the summed points by the highest achievable points per SR.
We used the Mann-Whitney U-Test to compare summary scores between both groups. At item level we have replaced the originally planned Chi-squared test by Fisher's Exact Test due to low cell numbers for some items. For all item-level analyses, we combined "yes" and "partial yes" into one category and in this case considered the item to be completely ful lled. P-values < 0.05 were considered to be statistically signi cant.

Study selection
The systematic literature search of the electronic databases identi ed 2968 references. After screening titles and abstracts, we retrieved full-text of 356 articles. Of these, 185 SRs met all inclusion criteria. Only 27 SRs were industry funded . After matching, 30 non-industry funded SRs (56-85) were randomly chosen resulting in a total sample size of 57 (see Fig. 1). The remaining 128 SRs were not considered in the analysis. A list of excluded references can be found in appendix 3.

Study characteristics
The selected 57 SRs comprised a total of 1294 studies (without considering overlap of studies between SRs). The median number of included studies was 15.0 (range 3-150). The median journal impact factor was 3.14 (range 1.18-24.45; n = 54, three articles were published in a journal without an impact factor).
The majority of SRs included non-randomized studies (44/57, 77%). Network meta-analysis was conducted in one study only (49). The SR interventions represented a broad range of vaccine-preventable infectious diseases dominated by in uenza (14/57, 25%), streptococcus pneumoniae (12/57, 21%), and human papillomavirus infection (8/57, 14%); and also including hepatitis, tetanus, diphtheria, pertussis, poliomyelitis, rotavirus, Haemophilus in uenza type b, tuberculosis, measles, mumps, rubella, and herpes zoster. The types of industry con ict for the "industry-funded" SRs included study funding by a vaccine manufacturer (3/27, 11%), and in most cases direct employment of at least one author by a vaccine manufacturer (24/27, 89%). All but two of these SRs did not report a con ict of interest. Out of the nonindustry funded SRs, 47% (14/30) reported no funding, while 43% (13/30) reported funding from nonpro t organizations or governmental/national agencies. Further study characteristics are shown in appendix 4.

Methodological quality
The mean AMSTAR 2 summary score across all SRs was 0.49. Overall, the median AMSTAR 2 summary score was higher for the non-industry funded SRs than for the industry-funded SRs (0.62 vs. 0.36; p < .00001). Industry funded SRs were less likely to ful ll all but one AMSTAR 2 item compared to nonindustry funded SRs (Fig. 2), though the difference was signi cant only for three speci c items (item 6: data extraction, item 14: assessment of heterogeneity, and item 16: con ict of interest. All AMSTAR ratings can be found in appendix 5.

Discussion
This is the rst study to focus on the in uence of industry-funding on the methodological quality of vaccine SRs. We found that overall methodological quality of SRs of vaccines was suboptimal.
Furthermore, industry-funded vaccine SRs over the last three years turned out to have a lower methodological quality than non-industry funded SRs.
The number of published SRs is rising steadily (9, 86). In our sample, the number of published SRs was rather balanced across 2016 to 2018, while there was a notable increase in 2019. Of all SRs that met our eligibility criteria, 15% (27/185) were industry funded. This can be considered a high proportion of industry funded SRs. In two recent samples of SRs in dentistry the proportion of industry-funded SRs was 2% (87), while it was 3% in a general sample of SRs (88). In a sample of SRs in vaccinology published between 1991 and 2007 the proportion of industry funded SRs was 7% (9/121) (89).
Overall, the methodological quality of the included SRs was low to moderate as indicated by a mean summary score of 0.49. We acknowledge that calculating a summary score is not the recommended approach to derive the methodological quality; nevertheless, others have already started using this intuitive approach (90). Furthermore, for the purposes of statistical comparative research-on-research studies such as ours this method allows for the comparison of two groups of SRs by ruling out potential oor effects that might arise with the original rating system (27). Other studies that used a comparable approach to derive the methodological quality revealed even lower scores of 0.19 in rehabilitation (91), while it was only slightly higher with 0.55 in a sample of acupuncture SRs (92). Interestingly, in contrast to our ndings, an older study using AMSTAR (the original version preceding AMSTAR 2) found SRs on in uenza vaccination to be of high methodological quality (93), while moderate methodological quality was recently found in a study dealing with SRs on interventions aimed at improving vaccination coverage (94).
To the best of our knowledge, the in uence of industry funding in SRs on vaccines has only been investigated in two studies. Both studies were not speci cally designed to analyze the impact of industry funding. In the rst study, Remschmidt et al. focused on SRs on in uenza vaccination only (93). The median AMSTAR score was higher for non-industry funded SRs but the difference did not reach statistical signi cance. Our present sample of SRs was not restricted to any speci c vaccine-preventable disease. One can speculate whether the difference in the samples between our broad analysis versus the targeted in uenza analysis can explain the different ndings. One potential explanation is that in uenza vaccines require complex summative analyses across seasons, and the large impact of the seasonal in uenza vaccine programs could drive a high quality effect. This would explain the high methodological quality of these SRs observed in the study by Remschmidt et al. (93). In the second study, De Vito et al. analyzed characteristics and methodological quality of SRs in vaccinology. They found that nancial support was associated with a higher methodological quality. However, they did not separate their analysis to investigate the in uence of industry funding, although they reported that SRs obtaining for-pro t funding had a slightly higher methodological quality than SRs with non-pro t funding (89). Overall, the comparability of this study might be hampered by the fact that they used the Overview Quality Assessment Questionnaire (OQAQ) to assess the methodological quality. The OQAQ was the only validated tool at the time of conducting the study (95), while it has been superseded by AMSTAR that draws on the OQAQ and lately AMSTAR 2 due to criticism and methodological improvements.
The other most commonly used critical appraisal tool for SRs is the Risk of Bias in Systematic Reviews (ROBIS) tool (96). While ROBIS and AMSTAR have been found to be equally valid (22,24,25), it is notable that ROBIS does not include a risk of bias domain focused on source of funding. Our ndings underscore the need to also consider source of funding.
Other studies we can compare our ndings to do not focus on vaccines but on pharmaceuticals. Although there might be some differences between vaccines and pharmaceuticals given some challenges that are mostly unique to vaccine SRs, there is overlap in the manufacturers of vaccines and pharmaceuticals. Several studies found that industry-funded SRs had lower methodological quality and had more aws in reporting (97)(98)(99). The most recent of these studies also highlighted that the quality of industry funded SRs improved over time (99).
At the item level, we found statistically signi cant differences for data extraction (item 6), assessment of heterogeneity (item 14), and con ict of interest (item 16). Item 6 and item 14 were not found to be a source of difference between industry and non-industry funded SRs in any of the other comparable studies. In particular, the difference in item 6 is somewhat surprising. However, not ful lling item 6 does not necessarily mean that data extraction was not performed in duplicate but could also mean that this was not su ciently reported. The developers of AMSTAR 2 have declared seven items to be critical (20). However, none of the three items we found a difference for is critical according to the developers of AMSTAR 2. A recent survey involving a ranking exercise also found these three items not to be highranked when compared to the remaining AMSTAR 2 items (100). We also experienced di culties when assessing item 16 for industry-funded SRs. To ful ll item 16 on con ict of interest, SR authors either have to declare that there were no competing interests which is obviously not an option for industry funded SRs, or to describe their funding sources and how they managed potential con icts of interest. However, we found no guidance on what constitutes an adequate way to manage con ict of interest in industry funded SRs. We only considered this to be ful lled by two SRs what might have been very strict given that no guidance is available.

Strength and limitations
Our study is able to provide an up-to-date picture of the number of methodological quality of SRs in the eld of vaccination. However, we restricted our sample to SRs dealing with interventions only as AMSTAR 2 has been designed for this type of SRs only, while it has not been validated for other types of SRs (e.g. etiology/risk factors, prevalence or incidence). Thus, all but one SR (including a network meta-analysis) were evaluated with a measurement tool that has been validated for this purpose, what has been found a methodological aw before (101). We are not certain that our ndings can be generalized and extrapolated to other types of SRs. To ensure consistency, all steps of study selection, data extraction and critical appraisal involved a calibration exercise and the involvement of an experienced reviewer (DP).
Thus, we are con dent that the approach of combining the results of multiple reviewers did not hamper the quality of our data. We originally also intended to look at spin but were unable due to feasibility issues.

Conclusion
The methodological quality of SRs in vaccination is comparable to SRs in other elds, while it is still suboptimal. Methodological quality of SRs needs to improve; which is a task for authors, reviewers and editors. In particular, decision-makers relying on SRs to inform health policy should be aware of methodological limitations of SRs. We are not able to provide a satisfactory explanation why industry funded SRs had a signi cantly lower methodological quality than non-industry funded SRs over recent years. This needs to be explored in future. Industry funding is an important indicator of methodological quality for vaccine SRs and should be carefully considered when appraising SR quality.

Declarations
Ethics approval and consent to participate Does not apply.

Consent for publication
All authors approved the last version of the manuscript.

Availability of data and material
All collected data can be found in the appendices.
Competing interests TH co-author of four included studies, not involved in assessment.

Funding
This research did not receive any speci c grant from funding agencies in the public, commercial, or notfor-pro t sectors.

Authors' contributions
All authors attest they meet the ICMJE criteria for authorship. IH, CB, GP, and SS used the gathered data partly for their bachelor theses.