Impact of risk of bias on the magnitude of treatment effects of randomized controlled trials in implant dentistry

Background: Risk of bias (RoB) could inuence the magnitude of treatment effects of randomized controlled trials (RCTs). This study aims to investigate the potential inuence of RoB on treatment effects estimates in RCTs in implant dentistry. Methods: The RCTs published in ve leading oral implant journals during the recent ve years were electronically searched. The RoB was assessed using the Cochrane Collaboration RoB tool. The meta-regression analysis and Monte Carlo permutation test were performed to identify the association between RoB and the magnitude of treatment effects. Results: A considerable amount of studies have high RoB in blinding of participants and personnel, and unclear RoB in allocation concealment and selective reporting. The treatment effects were exaggerated by aws in allocation concealment for binary outcomes and by deciencies in random sequence generation and selective reporting for continuous outcomes. Conclusion: RoB frequently exists in RCTs recently published in implant dentistry, which may lead to the exaggeration of treatment effects. Better study design, implementation, and reporting are required for clinical trials in implant dentistry to ensure more reliable evidence.


Background
Random controlled trials (RCTs) are the most credible assessment of treatment interventions among various study designs. RCTs, systematic reviews, and meta-analyses based on RCTs can provide clinicians with guidance [1,2]. However, the quality of the study often in uences the reliability of RCTs [3,4]. For example, the lack of concealment of randomized allocation elevates the risk that patients with better prognosis are assigned to the preferred group, and the lack of blinding to outcome assessor is likely to exaggerate the bene cial effects of treatment interventions [5]. It is well recognized that the bias occurring in the design and conduct of studies could impair the internal validity [6], and induce systematic deviations in the estimation of intervention effects of RCTs [7].
Unfortunately, the quality of the RCTs in dentistry was suboptimal [8]. Risk of bias (RoB) has been quite common in RCTs in implant dentistry, especially [9]. The bias can occur during the whole course of RCTs. Nowadays, we can use a series of tools to evaluate the RoB in medical researches comprehensively [10].
The Cochrane Collaboration RoB tool that assesses the bias from 6 perspectives (selection bias, performance bias, detection bias, attrition bias, reporting bias, and other sources of bias) in 7 domains is most widely used in appraising RCTs [11].
An increasing amount of studies have recently explored the association between the RoB and the estimated treatment effects of RCTs. Some studies found that RoB in RCTs could lead to overestimation or underestimation of treatment effects [4,12,13], while others reported negative results [14]. A study focusing on RCTs in orthodontics suggested that RCTs with high RoB in the domains of random sequence generation and incomplete outcome data had found exaggerated treatment effects [15].
Nevertheless, another study detected no association between RoB and effect sizes in periodontal literature [16]. Thus, whether the RoB causes deviation in estimated effects remains a question.
The objectives of this study are (1) to evaluate the RoB of RCTs in implant dentistry, and (2) to investigate the impact of RoB on the estimated treatment effects in implant dentistry.  (Table S1) [17].

Study selection
The records identi ed through electronic searching were evaluated according to the title, abstract, and full-text when needed. Only RCTs within implant dentistry were included in this study. The nonrandomized researches, clinical researches on other specialty, and nonclinical studies were excluded. Two authors independently assessed studies for inclusion. Disagreements were resolved through a discussion with a third author.

Data extraction
Data extraction from included RCTs was conducted by two authors independently. The basic characteristics and estimated treatment effects were extracted. The characteristics included the name of the rst author, the name of the journal, year of publication, the continent of origin (based on the corresponding author), number of authors, number of study centers, the involvement of statisticians, availability of informed consent and sample size. The estimated treatment effects were mean value and the standard deviation (SD) for continuous outcomes and the number of events and non-events for binary outcomes, respectively.

Risk of bias assessment
The Cochrane Collaboration RoB tool was used to appraise the RoB of included RCTs in 7 domains (random sequence generation, allocation concealment, blinding of participants and personnel, blinding outcome assessment, incomplete outcome data, reporting bias and other sources of bias) by two authors independently. The Cochrane Collaboration provides guidelines to score each item. Each domain is evaluated as 'yes' (low RoB), 'no' (high RoB) or 'unclear' (unclear RoB) [11]. Moreover, Cohen's kappa statistics was adopted to assess the interrater agreement before the start of the RoB assessment. Brie y, the assessment results of the RoB of 20 selected studies by two authors were used for calibration. The selected studies were a subset of included studies that were randomly selected by the research randomization software (www.randomizer.org).

Statistical analysis
To analyze the impacts of RoB on treatment effects for each domain, we used an approach previously described [15]. Binary and continuous studies were analyzed separately to reduce the heterogeneity introduced by the different types of outcomes [18]. Speci cally, the effects estimates were expressed as odds ratio (OR) and standardized mean difference (SMD) for binary and continuous datasets, respectively. For each domain, unclear and high RoB were combined to convert into a dichotomous variable according to the recommendation from the Cochrane Handbook [19]. Low RoB was used as the reference level. Subsequently, single covariate meta-regression analyses of each domain were conducted. The results of RoB assessment and the effect estimates were regarded as the independent and dependent variables severally. Besides, the essential characteristics of RCTs, including journal name, the continent of origin, study type, number of authors and centers, the requirement of informed consent and involvement of statisticians, were also analyzed as confounding factors for effect estimates using metaregression models. Monte Carlo permutation tests were adopted for each meta-regression model to reduce the risk of false-positive results caused by type error [20]. A two-tailed p-value < 0.05 was considered statistically signi cant with 95% CI. All analyses were conducted using Stata™ version 15.0 (Stata Corp LP, College Station, TX, USA).
As for a multi-arm RCT, the control group was split into more groups with smaller sample size, which were then compared with other experimental groups respectively to avoid the error of in ating sample size during statistical analysis [15]. For instance, a three-arm RCT was converted into two pair-wise comparisons by dividing the control group into two control groups with smaller sample size. For binary outcomes, both the number of events and total participants were split. For continuous outcomes, the means and standard deviations were kept as reported [19].

Characteristics of the included studies
Through the process of study selection, 230 studies were considered eligible for the current study from 516 initially retrieved records (Fig. 1). Among 35 studies with binary outcomes, ve studies were threearm trials. Similarly, eighteen studies adopted a three-arm design for treatments among 195 studies with continuous outcomes, whereas nine studies carried out four-arm trails. Eventually, a total of 271 pair-wise comparisons were involved in the statistical analysis. The distribution of the characteristics of the included studies was shown in Table 1. The majority of included studies obtained informed consent from participants (203, 88.3%) and were conducted in a single center (194, 84.3%). Most of them were of European authorship (133, 57.8%) and involved 5-6 listed authors (129, 56.1%). A considerable amount of studies was published in CIDRR (69, 30.0%). While only 62 studies (27.0%) reported that a statistician supported the study design or statistical analysis.  The Monte Carlo standard error of the adjusted p-value ranged from 0.0007-0.0050, indicating that the number of permutation tests was enough to obtain accurate results.  As for the results of meta-regression, there is no association between basic characteristics and the treatment effect sizes in studies with continuous outcomes (Table 3). However, the RCTs tended to report overestimated intervention effect estimates when unclear/high RoB existed in random sequence generation (SMD: 0.364, 95%CI: 0.041-0.687, p: 0.025) and selective outcome reporting (SMD: 0.356, 95%CI: 0.004-0.708, p: 0.044)( Table 4). The Monte Carlo standard error of p-value ranged from 0.0015-0.0050, which ascertained the strength of permutation tests.

Discussion
In this study, we assessed the RoB in RCTs published in the leading journals in implant dentistry during the recent ve years using the Cochrane collaboration RoB tool. The RoB assessment presented that most studies with high RoB were due to lack of blinding of participants and personnel and other sources of bias. At the same time, unclear RoB existed in the process of allocation concealment and selective reporting. The domains evaluated mostly with low RoB were random sequence generation, blinding of outcome assessment, and incomplete outcome data.
Unlike the RCTs on drug therapy and many other elds of medicine, blinding of participants and personnel, which inclines to be assessed as high RoB, is often inapplicable in clinical trials in implant dentistry on account of the visually distinguishable interventions [11]. Failure of blinding to personnel may result in a more careful and elaborate operation to an individual who is presumed to have a better prognosis. At the same time, the lack of blinding to participants may in uence their compliance and subjective outcomes such as Oral Health-related Quality of Life.
Similar to previous analyses [8,15,21], the domains mostly assessed with unclear RoB in this study were allocation concealment and selective reporting. RCTs are usually assessed as unclear RoB due to the short description of the methodology [11]. The detailed reporting of clinical trials is the prerequisite to evaluate the internal validity accurately [9]. Consolidated Standards of Reporting Trials (CONSORT) statement is the most frequently endorsed reporting guidelines for RCTs in dental journals [22]. However, only two of the selected journals (CIDRR and COIR) have required CONSORT or any other reporting guidelines according to "Author Guidelines". Authors and journals need to adhere to reporting guidelines such as CONSORT more strictly for more accurate RoB assessment of RCTs in implant dentistry.
In addition to the assessment of RoB, we also analyzed the associations between RoB and treatment effects. Previous studies reported that treatment effects could be more exaggerated of the RCTs with unclear/high RoB in random sequence generation and allocation concealment [4,5], which is consistent with our ndings. The random sequence generation and allocation concealment are both belong to selection bias [23]. The unclear/high RoB of selection bias could result in heterogeneity between control and experimental groups and further, misguide clinical practice. Under this circumstance, treatment effects are the re ection of differences from baselines to a certain degree rather than differences from interventions alone [7,24]. Therefore, the decisions based on RCTs in implant dentistry with de ciencies in these two domains should be adopted with caution.
Furthermore, we identi ed the association between the overestimated treatment effects and unclear/high RoB in selective reporting. Reporting bias arises when the outcomes are reported selectively for publication [25]. Generally, statistically signi cant outcomes are more likely to be reported than those with negative results [26]. In this study, most of the RCTs assessed unclear/high RoB in selective reporting were attributed to the lack of information on the protocol registry. This nding demonstrates the reporting bias is a threat to the validity of RCTs in implant dentistry and encourages the future clinical trials to be registered at the public database and reported as de ned by the trial protocol.
Apart from RoB, the lack of statisticians as one of the confounding factors caused the overestimation of treatment effects in studies with binary outcomes as well. To the best of our knowledge, our study is the rst to identify a positive association between the lack of statisticians and biased intervention estimates in the literature of a dental specialty. The importance of recruiting statisticians into medical researches has been well recognized [27]. Previous analyses found that more than half of publications [28, 29] had at least one methodological error. The involvement of a statistician is bene cial for an appropriate adjustment of statistical analysis and might decrease the RoB [30].
Based on the current study, there are some recommendations. Firstly, when clinicians interpret the results of RCTs that have unclear/high RoB in the sections of selective and reporting bias in implant dentistry, particular caution is needed. Secondly, the implementation and report of an RCT should strictly follow validated reporting guidelines such as CONSORT. Meanwhile, it is suggested that statisticians should be involved in the design, analysis, and report of clinical trials. At last, we recommend that journals take a more proactive approach to ensure the adherence of RCTs in implant dentistry to reporting guidelines.
Some limitations of the study may restrict the generalization of the results. For example, we only searched ve journals in the eld of implant dentistry. Thus the comparatively small number of RCTs included in this study could provide researchers with meaningful but not broadly generalizable guidance.
Moreover, the assessments could be, to some degree, susceptible to subjectivity even though the RoB assessment of included studies was determined according to the Cochrane Collaboration RoB tool strictly.

Conclusions
High RoB in blinding of participants and personnel and unclear RoB in allocation concealment and selective reporting existed in RCTs in implant dentistry. Unclear/high RoB in random sequence generation, allocation concealment, and selective reporting were likely to cause the overestimation of treatment effects. The involvement of a statistician in the trial design or statistical analysis could alleviate the risk of result exaggerations. Therefore, the researchers in implant dentistry should design and conduct RCTs with more attention in the processes of random sequence generation, allocation concealment, and selective reporting with the help of statisticians. Author contributions: RD and WL made substantial contributions to the conduction of the work, the analysis of the data and the drafting of the manuscript. JY and LZ made substantial contributions to the design of the work, drafting and revision of the manuscript. LZ and ZZ supervised the whole study and made substantial contribution to the concept of the study and critical revision for the important intellectual content. All authors read and approved the nal manuscript.  Figure 1 The ow diagram of the study selection.