This is the first study to focus on the influence of industry-funding on the methodological quality of vaccine SRs. We found that overall methodological quality of SRs of vaccines was suboptimal. Furthermore, industry-funded vaccine SRs over the last three years turned out to have a lower methodological quality than non-industry funded SRs.
The number of published SRs is rising steadily (9, 86). In our sample, the number of published SRs was rather balanced across 2016 to 2018, while there was a notable increase in 2019. Of all SRs that met our eligibility criteria, 15% (27/185) were industry funded. This can be considered a high proportion of industry funded SRs. In two recent samples of SRs in dentistry the proportion of industry-funded SRs was 2% (87), while it was 3% in a general sample of SRs (88). In a sample of SRs in vaccinology published between 1991 and 2007 the proportion of industry funded SRs was 7% (9/121) (89).
Overall, the methodological quality of the included SRs was low to moderate as indicated by a mean summary score of 0.49. We acknowledge that calculating a summary score is not the recommended approach to derive the methodological quality; nevertheless, others have already started using this intuitive approach (90). Furthermore, for the purposes of statistical comparative research-on-research studies such as ours this method allows for the comparison of two groups of SRs by ruling out potential floor effects that might arise with the original rating system (27). Other studies that used a comparable approach to derive the methodological quality revealed even lower scores of 0.19 in rehabilitation (91), while it was only slightly higher with 0.55 in a sample of acupuncture SRs (92). Interestingly, in contrast to our findings, an older study using AMSTAR (the original version preceding AMSTAR 2) found SRs on influenza vaccination to be of high methodological quality (93), while moderate methodological quality was recently found in a study dealing with SRs on interventions aimed at improving vaccination coverage (94).
To the best of our knowledge, the influence of industry funding in SRs on vaccines has only been investigated in two studies. Both studies were not specifically designed to analyze the impact of industry funding. In the first study, Remschmidt et al. focused on SRs on influenza vaccination only (93). The median AMSTAR score was higher for non-industry funded SRs but the difference did not reach statistical significance. Our present sample of SRs was not restricted to any specific vaccine-preventable disease. One can speculate whether the difference in the samples between our broad analysis versus the targeted influenza analysis can explain the different findings. One potential explanation is that influenza vaccines require complex summative analyses across seasons, and the large impact of the seasonal influenza vaccine programs could drive a high quality effect. This would explain the high methodological quality of these SRs observed in the study by Remschmidt et al. (93). In the second study, De Vito et al. analyzed characteristics and methodological quality of SRs in vaccinology. They found that financial support was associated with a higher methodological quality. However, they did not separate their analysis to investigate the influence of industry funding, although they reported that SRs obtaining for-profit funding had a slightly higher methodological quality than SRs with non-profit funding (89). Overall, the comparability of this study might be hampered by the fact that they used the Overview Quality Assessment Questionnaire (OQAQ) to assess the methodological quality. The OQAQ was the only validated tool at the time of conducting the study (95), while it has been superseded by AMSTAR that draws on the OQAQ and lately AMSTAR 2 due to criticism and methodological improvements.
The other most commonly used critical appraisal tool for SRs is the Risk of Bias in Systematic Reviews (ROBIS) tool (96). While ROBIS and AMSTAR have been found to be equally valid (22, 24, 25), it is notable that ROBIS does not include a risk of bias domain focused on source of funding. Our findings underscore the need to also consider source of funding.
Other studies we can compare our findings to do not focus on vaccines but on pharmaceuticals. Although there might be some differences between vaccines and pharmaceuticals given some challenges that are mostly unique to vaccine SRs, there is overlap in the manufacturers of vaccines and pharmaceuticals. Several studies found that industry-funded SRs had lower methodological quality and had more flaws in reporting (97–99). The most recent of these studies also highlighted that the quality of industry funded SRs improved over time (99).
At the item level, we found statistically significant differences for data extraction (item 6), assessment of heterogeneity (item 14), and conflict of interest (item 16). Item 6 and item 14 were not found to be a source of difference between industry and non-industry funded SRs in any of the other comparable studies. In particular, the difference in item 6 is somewhat surprising. However, not fulfilling item 6 does not necessarily mean that data extraction was not performed in duplicate but could also mean that this was not sufficiently reported. The developers of AMSTAR 2 have declared seven items to be critical (20). However, none of the three items we found a difference for is critical according to the developers of AMSTAR 2. A recent survey involving a ranking exercise also found these three items not to be high-ranked when compared to the remaining AMSTAR 2 items (100). We also experienced difficulties when assessing item 16 for industry-funded SRs. To fulfill item 16 on conflict of interest, SR authors either have to declare that there were no competing interests which is obviously not an option for industry funded SRs, or to describe their funding sources and how they managed potential conflicts of interest. However, we found no guidance on what constitutes an adequate way to manage conflict of interest in industry funded SRs. We only considered this to be fulfilled by two SRs what might have been very strict given that no guidance is available.
Strength and limitations
Our study is able to provide an up-to-date picture of the number of methodological quality of SRs in the field of vaccination. However, we restricted our sample to SRs dealing with interventions only as AMSTAR 2 has been designed for this type of SRs only, while it has not been validated for other types of SRs (e.g. etiology/risk factors, prevalence or incidence). Thus, all but one SR (including a network meta-analysis) were evaluated with a measurement tool that has been validated for this purpose, what has been found a methodological flaw before (101). We are not certain that our findings can be generalized and extrapolated to other types of SRs. To ensure consistency, all steps of study selection, data extraction and critical appraisal involved a calibration exercise and the involvement of an experienced reviewer (DP). Thus, we are confident that the approach of combining the results of multiple reviewers did not hamper the quality of our data. We originally also intended to look at spin but were unable due to feasibility issues.