This study aimed to revise Criterion II of the CQS-2 trial appraisal tool. To establish an empirical basis for this revision, we first tested whether meta-analyses of trials with inadequate allocation concealment generate positive test results. One meta-analysis with positive test results could be identified from which we logically deduced the criterion for adequate allocation concealment and, in accordance with its verbatim text, subsequently formulated the new Criterion II.
During our review, we excluded 43 out of 44 meta-analyses as unsuitable for testing. Therefore, the results of the single accepted meta-analysis [14] provide only a limited empirical basis for revising CQS-2/Criterion II. Furthermore, it was surprising that three of the five trial reports [15, 17, 19] labelled by the authors of the meta-analysis [14] as adequate allocation concealment did not contain information in that regard. We assumed that Kim et al. obtained this information by contacting the trial authors after these trial reports had been published. However, for the purpose of our study, we could thus rely, in addition to the meta-analysis text [14], on two trial reports [16, 18] only.
Despite these limitations, our study could establish the revised Criterion II (CQS-2B) on a more precise empirical basis than for the original CQS version (CQS-2). While the latter relied on the data and wording of meta-epidemiological evidence [5, 8], the former is based on the data and wording of one meta-analysis [14] from which the meta-epidemiological evidence [8] was established. Further precision might be achieved if individual randomised trials themselves could be tested for potential selection bias instead of meta-analyses of such trials. However, to our knowledge, only two test methods are currently available for this purpose, namely baseline testing within a trial and the Berger-Exner test. While baseline testing in individual trials may give some indication for allocation problems [20], it may generate misleading findings [10, 21]. The highly accurate Berger-Exner test [22] relies on raw data that are mostly accessible to the trial’s authors only, and thus can be conducted (and its results reported) by only the trial authors themselves. In contrast, the bias test presented by Hicks et al. [10] enables application by reviewers to empirically ascertain whether the results of meta-analyses of trials are biased. Since such meta-analyses also include the appraisal of bias risk, for example whether allocation concealment in trials was adequate or not, the wording of their appraisal criteria is more precise in relation to empirical bias test findings than the wording from meta-epidemiological studies that pooled several meta-analyses with differences in the wording of each of their appraisal criteria.
It has been suggested that basing trial appraisal criteria on empirical results from meta-epidemiological studies may be futile because most of these studies can control only incompletely for confounding and, subsequently, their results cannot be ascribed to the lack or incomplete reporting of a particular trial’s characteristics. Instead, the reliance on theoretical justification was suggested [23].
However, the sole theoretical justification lacks information on whether the theory corresponds with empirical facts. When a theory is compared with empirical facts, such comparison constitutes a test. If the test outcome is negative, then empirical facts are shown to contradict the theory, which in turn is then considered falsified. Such falsification is sufficient not to accept the theory. If the test result is positive, the hypothesis is considered corroborated. This does not mean that the theory is true but only indicates that it has passed the test for now and there is thus no current reason to reject it. As long as the theory remains corroborated, it can explain reality well and does not conflict with empirical facts for the time being.
In our study, we could establish from the limited available data, so far, that meta-analyses results of pooled trials with inadequate/unclear allocation concealment correspond with positive test results in terms of in-between-trial heterogeneity (I2 > 0) due to imbalances in the baseline variable ‘age’ between randomised groups. From these results, we logically deduced criteria for adequate allocation concealment, relating to the need for central allocation of patients according to the random allocation sequence.
The findings of our study are in line with past and current guidelines for assessing risk of bias in randomised trials by the Cochrane Collaboration. The Cochrane Reviewers’ Handbook version 4.2.1 stated in 2003 already that “the ideal is for the process to be impervious to any influence by the individuals making the allocation. This will be most securely achieved if an assignment schedule generated using true randomisation is administered by someone who is not responsible for recruiting subjects, such as someone based in a central trial office or pharmacy.” [24]. In 2006, the handbook’s version 4.2.6 maintains that “centralised (e.g. allocation by a central office unaware of subject characteristics) or pharmacy-controlled randomization” is one of the approaches that can be used to ensure adequate concealment [25]. From 2017 until now (2022), subsequent handbook versions have maintained that “central randomization by a third party is perhaps the most desirable” [26–32].
Recommendations For Further Research
The CQS is still in development. Prospective, controlled clinical therapy trials from systematic reviews that have applied the 2nd version of Cochrane’s RoB tool may be re-appraised using the new CQS-2B version to establish whether the direction and magnitude of any pooled effect estimates remain the same. Based on the results of these further investigations, the CQS-2B may be piloted as part of the regular, systematic review methodology for the appraisal of prospective, controlled clinical therapy trials.