In this study, using open websites as the reference, we observed an underreporting among one third of systematic reviews disclosing no financial conflict of interest by the authors. It was estimated that in all half of all systematic reviews were associated with financial conflict of interest. Most systematic reviews also had high risk of bias, and the proportion not compromised by either financial conflict of interest or high risk of bias was negligible. These findings are discouraging for readers of systematic reviews on the effects of methylphenidate for patients with ADHD.
To the best of our knowledge, we have not met with a study adopting a mode of procedure similar to ours for investigating the accuracy of financial conflict of interest disclosed in systematic reviews. A procedure somewhat similar to ours, comparing disclosures of financial conflict of interest by authors of 281 reviews with their other publications, was reported in a systematic meta-review published more than ten years ago focussing on new drugs for the treatment of arthritis [9]. The study reported 16% of the reviews had authors that had published several reviews at during a reasonably short time period but only disclosed conflict of interest in one of them. It was also found that 22% of the reviews had authors that failed to disclose their participation in a clinical trial of the same drugs. The authors noted conflict of interest was reported less frequently in narrative reviews.
A study investigating guidelines on drugs from across 38 specialist societies used a disclosure list kept by the health authorities as the reference standard [10]. This study reported 96% of guidelines included at least one author with conflict of interest.
A recent systematic survey reported findings from primary studies that assessed the concordance of disclosures of financial conflict of interest with databases of payments for 27 included documents including both published articles, guidelines and meeting disclosures [11]. The range of completely non-concordant’ disclosures was median 43% (range 15–89%). However, the reviewers also pointed out that the methodological quality of several included studies was poor. Despite differences in the methodological approach underreporting is not a deviant finding.
Our methodology of searching for financial conflict of interest in publically available open websites was also validated by the confirmatory findings for authors of 15 systematic reviews who had disclosed financial conflict of interest. We used the same proceedings for all searches, we only turned to open websites that authors/researchers control themselves, and our approach was deliberately conservative. We noticed that information on some authors, probably at the beginning of their career, was more difficult to retrieve. Common names turned out to be an underestimated problem, especially if there was no English version of the homepage. In other words, it can not be ruled out that some of the included systematic reviews were falsely classified as negative.
We found at least one author disclosed conflict of interest in one third of systematic reviews and this matches previous findings. In 2016, Hakoum et al reported on the difference between disclosures of conflict of interest in SR by published by Cochrane versus others [3]. The proportion of disclosures for non-Cochrane reviews was 30%, and for Cochrane it was 49%. An investigation of systematic reviews on sugar-sweetened beverages and weight gain or obesity estimated the rate of disclosed financial conflict of interest to 35% [12], whereas a study on systematic reviews on neuraminidase inhibitors reported financial conflict of interest for at least one author in 22% [13].
It was a disappointing experience that we were not able to determine the direction of conflict of interest for most SR, as we expected this would be straightforward once conflict of interest was encountered. However, several of the included systematic reviews were complex and evaluated many interventions simultaneously. Moreover, disclosures reveal an association with a company or organisation but not with a specific product or drug. For primary studies, it seems possible to assess the direction of conflict of interest more easily, as s conflict of interest has been associated with a more favourable result [14]. In our data on SR, the direction of conflict of interest was, with very few exceptions, not possible to discern. Nevertheless, conflict of interest has been associated with more favourable conclusions of systematic reviews [2, 15].
An overwhelming majority (84%) of the included systematic reviews were associated with a high risk of bias. The most common reasons were suboptimal identification and selection of studies for almost half, and unsatisfactory definitions of study eligibility criteria in almost one third of the included systematic reviews. This is an improvement compared to the findings of Roundtree more than ten years ago, who reported merely 5% of SR explained search criteria explicitly, or used a criterion-based selection of studies [9]. Several tools for assessment of risk of bias /study quality have been developed over time such as QUROM [6], the Cook et al score[16], AMSTAR [17], AMSTAR-2 [18], and lately ROBIS introduced in 2016 [19]. Nevertheless, a large proportion of systematic reviews are estimated to be associated with high risk of bias, or low quality[20–23]. A very recent example is an investigation of 48 SR on COVID-19, only 5/48 (10%) were found to be at low risk for bias using the ROBIS tool [20]. Such large proportion at high risk of bias dispute the notion of systematic reviews as the top of the pyramid or hierarchy of evidence. It also highlights the importance of a qualified peer review process of systematic reviews. The general recommendation of always approaching a new clinical topic by reading the systematic reviews first has to be supported by increased awareness and training to spot methodological shortcomings.
The numbers of included systematic in this investigation are too small for any statistical evaluation but, interestingly, we found a larger proportion of SR with conflict of interest had a low risk of bias compared to those without conflict of interest (6/24 vs 1/20, Table 3). This is contrary to any supposition that conflict of interest is associated with high risk of bias/low quality. Our findings is in agreement with of Lundh et al who investigated primary studies and reported that, except for blinding, there were no differences in risk of bias between industry sponsored studies and those without such conflict of interest [14]. This underlines the importance of assessing potential bias by conflict of interest separately, apart from the methodological risk of bias assessment. It can no longer be taken for granted that risk of bias by financial conflict of interest will be included or detected by the tools for risk of bias assessment commonly used today.
According to the PRISMA guidelines, conflict of interest of primary studies included in a systematic review should be reported in the SR [7]. In our sample, only 8/44 (18%) SR reported information on conflict of interest of the included primary studies. This is similar to an investigation of non-Cochrane reviews in which 15% of SR had reports on funding for the included primary studies [24]. Conflict of interest among authors of primary studies may also be associated with tactical choices on control groups or comparative interventions [25]. The poor attention to this crucial aspect of systematic reviews is an urgent topic for further studies.
A limitation of this study was the focus on merely financial conflict of interest. A recent update of Cochrane´s policy point out the importance of also non-financial conflict of interest [26]. Advantages of our investigation are the search for financial conflict of interest at open websites carried out by two independent, blinded reviewers and a meticulous assessment of risk of bias. Concerning the impossibility to determine the direction of conflict of interest, and analyse any association with the results of systematic reviews, we urge declarations of conflict of interest be taken one step further, in particular for systematic reviews. To be meaningful, a disclosures must comply with the COPE (Committee On Publication Ethics) recommendations and competing interests should be clearly and consistently indicated, with links to further details if necessary [5]. Disclosures of conflict of interest without any opportunity of understanding the direction of the competing interests, at least not for the majority of readers, is not very helpful.
In summary, this meta-study on systematic reviews investigating Methylphenidate for treatment of ADHD have identified comprehensive shortcomings on the disclosure of financial conflict of interest and on the methodology used, including the reporting on conflict of interest of included primary studies.