Quality Assessment of Clinical Trials Included in Cochrane Oral Health Systematic Reviews

Background: Risk of Bias (RoB) and other characteristics of randomised clinical trials included in Cochrane oral health systematic reviews were assessed. Methods: All the trials included in Cochrane oral health systematic reviews were identied and examined. The RoB was evaluated for all the included clinical trials according to the Cochrane review standards. The Overall Risk of Bias (ORoB) was dened in this study based on the criteria in Cochrane’s RoB tool-v2. Descriptive analyses were carried out to determine the frequency of each intended variable. Results: In a total of 2565 included studies, the majority (n=1600) had 50 or higher sample sizes. As for blinding, 907 studies were labelled as double-blind. Performance bias showed the highest rate of high risk (31.4%). Almost half of the studies had a high ORoB compared to 11.1% with low ORoB. The studies that used placebos had higher low ORoB (14.8% vs 10.7%). The double-blind studies had the highest low ORoB (23.6%). The studies with a cross-over design had the highest low ORoB (28.8%). Conclusion: The RoB of oral health studies in Cochrane reviews was deemed high. Special efforts may be required to improve conducting and reporting of trials in this area.


Background
The quality assessment of studies should always consider both internal and external validities (1), which are critical aspects of any scienti c project, although internal validity is more relevant to empirical studies (2). There were some efforts to improving the quality of reporting of randomised controlled trials (RCTs); for example, Consolidated Standards of Reporting Trials (CONSORT) statement which was rst introduced in 1996 and then updated till 2010 (3). Furthermore, the Preferred Reporting Items for Randomised Trials in Endodontics (PRIRATE) 2020 guidelines have been developed by modifying and integrating the CONSORT guidelines speci cally for Endodontics (4). The Risk of Bias (RoB) is also a good measure of a study's internal validity (5,6). Bias in clinical studies leads to the over-or underestimation of treatment outcomes (7). From a clinical perspective, bias can lead to the application of an intervention that may not be effective or that may even be potentially harmful (6). While RCTs are widely considered the gold standard for therapeutic clinical research to measure new medical interventions' effectiveness, they are also prone to bias (8).
The Cochrane Database of Systematic Reviews is proposed as the biggest and most recognised scienti c database of systematic reviews and meta-analyses in health sciences that publishes high-quality systematic reviews with speci c methodological protocols (9). In the case of RoB, Cochrane reviews use a speci c domain-based evaluation tool that has been changed over the past years, with the rst edition introduced in 2008 (10). Currently, the domains consist of 'sequence generation', 'allocation concealment', 'blinding of participants and personnel', 'other potential threats to validity (other sources of bias)', 'blinding of outcome assessment (blinding of the examiner)', 'incomplete outcome data', and 'selective outcome reporting'. Cochrane's evaluation tool for RoB is a signposting scale with three variations, namely, 'Low-risk', 'High-risk' and 'Unclear-risk'. The rst two variations are self-explanatory. The unclearrisk choice is assigned when there is not enough evidence available (reported by the study) to explicitly conclude any speci c domain. The primary sources of support and reliance for Cochrane judgments are evidence or quotes from the appraised paper or any rational inferences and reviews drawn from the author's explanations (11).
Despite all the invaluable efforts made in the past few years, there is still a considerable gap between the available evidence and dentistry's common clinical procedures. As a result, in many cases, it may be di cult for practitioners to extract their needed data from the relevant studies (12). Moreover, there are also studies with inadequate allocation concealment and high RoB, which tend to change empirical results in a clinical setting (13). Without a way to offer dentists high-quality evidence to reinforce their decision-making process, dentists will face substantial stress, and their decision-making will become more time-consuming in clinical settings.
Previous studies with different aims tried to provide a picture of the current RoB in dentistry and oral health literature. However, some of these studies have focused on a speci c topic (14,15), assessed systematic reviews rather than RCTs (16), or used a sample rather than including all relevant RCTs (11,17). The present study aimed to assess the RoB and other characteristics of the RCTs included in Cochrane oral health systematic reviews about dentistry and oral health. Further suggestions to be considered by future Cochrane review teams will be presented toward the end.

Methods
This study is a secondary analysis of the clinical trials included in Cochrane oral health systematic reviews. Cochrane review authors have made all the judgements concerning the quality of the RCTs.

Search strategy
The "dentistry and oral health" lter was used to search Cochrane Library's "Search Reviews" section on June 1, 2020. All the reviews in this category were included in the data extraction step.

Identi cation and selection of primary studies
The primary studies were identi ed using the references included in the most updated Cochrane reviews, and their data were extracted by the characteristics of the included studies' tables. We excluded the reviews that only included observational studies. Figure 1 shows the detailed ow diagram of steps taken to choose the eligible studies.

Data extraction and management
This phase of the study was carried out for all the included trials by seven independent authors (AS-M, PI, MS, ES, HM, SN, PM), and the results were entered directly into preformatted Excel 2016 spreadsheets. In order to calibrate the authors, they were all trained regarding the protocol by extracting the information from 50 random RCTs as a pilot. All the spreadsheets were then merged into one le and rechecked to avoid any mistakes and inconsistencies among the data extractors (AS-M, PI, MS). The items intended to be extracted were divided into three categories: (1) general information of Cochrane reviews including the type of review, the subject, the number of included trials; (2) basic information on the RCTs, e.g. title, rst author, year of publication, journal, etc.; (3) the information reported in the RCTs (the "Characteristics of the included studies" section), which characterised the main methodological features of each RCT; and (4) the RoB assessment of the RCTs. Additionally, using the rst authors' a liations, we assessed the distribution of the RCTs' locations. We also evaluated the Cochrane reviews' subjects (de ned by Cochrane Library's classi cation for dentistry and oral health) with the highest citations according to the Web of Science. The mean number of citations per subject was measured, as well.
The following characteristics in the journals with more than three RCTs in the Cochrane reviews were assessed as well: their ranking, subject (according to SCImago Journal Rank 2019), and the number of inclusions in the Cochrane reviews.
Given that the Cochrane review authors evaluated the blinding process differently, the RoB assessments were performed by different trajectories. In some studies, blinding took place in two subgroups, i.e. blinding of the participants and personnel (performance bias) and blinding of the outcome assessment (detection bias); meanwhile, in some other studies, blinding was taken as one item. In rare instances, blinding took place in two categories, including subjective and objective outcomes or a range, namely, assessor, analyst, participants, and caregivers. In these cases, an aggregated form was provided; if one or more subcategories were reported as high-risk, the whole category of blinding bias was considered highrisk, and if all were at low risk, the whole category was considered low-risk; otherwise, the blinding bias was reported as unclear.
Some other RoB assessment items were rarely reported; for instance, funding bias, intention to treat bias, sample size bias, for-pro t bias, and power calculation bias. Due to these items' low frequency, these kinds of biases were not extracted in this study. Additionally, a new variable was de ned to aggregate the assessments' full results, named the "Overall Risk of Bias" (ORoB). The criteria for determining the ORoB were the same as those of the "Overall Bias" assessment in Cochrane's RoB tool -version 2 (18). If any of the bias domains were considered high-risk, the ORoB was considered high. In the absence of high-risk biases, unclear-risk-assessed biases were perceived in the same way; otherwise, the ORoB was considered low (19). The RCTs that appeared in more than one Cochrane review were excluded if their RoB assessments were not consistent.
In including RCTs in this study, several groups were considered, namely, quasi-experimental (controlled before-after, interrupted time series, and non-randomised designs), parallel RCT, cross-over RCT, cluster RCT, split-mouth RCT, and repeated-measures study designs.
Finally, to scrutinise the Cochrane review authors' subjectiveness and bias of judgement, the studies were distinguished by RoB domains. Cochrane reviews that included the same studies, either with the same or different RoB assessments, were listed and signposted in different categories based on the included titles.

Statistical analysis
The extracted data were analysed using Microsoft Excel (Microsoft O ce Professional, 2019). Mean and standard deviations were reported for the quantitative continuous variables and frequencies for the qualitative data.

Characteristics of Cochrane Reviews
Of the 8293 Cochrane reviews on the subject, 203 (2.4%) were on dentistry and oral health (23rd rank among the subjects in terms of the number of reviews). The majority of the reviews were interventional (99.0%), and there were two diagnostic studies. The review subjects were mainly concerned with dental caries (24.6%), craniofacial anomalies (18.7%), and oral and maxillofacial surgery (15.8%). Seventeen reviews (8.5%) included zero studies and 33 (17.9%) reviews were withdrawn for different reasons, mainly because of being out-of-date (75.8%). Nonetheless, ten of the studies withdrawn by Cochrane were included as they were only out-of-date and were still methodologically sound based on the latest Cochrane standards. Besides, two reviews included only observational studies, and we excluded them. Hence, 159 interventional reviews provided the data needed for this study.
The median of the number of studies included in these reviews was 10 (range: 1-154). Four reviews (4/159, 2.5%) had more than 100 included studies, and 80 (80/159, 50.3%) had less than ten (excluding the zero-including studies). The subjects of three most-cited reviews were oral cancer and pre-cancerous lesions (number of citations received = 309), oral and maxillofacial surgery (n = 305), and dental caries (n = 275) according to Web of Science. Nonetheless, reviews about antibiotic therapy (78.5, n = 2), oral lichen planus (73.5, n = 2), and periodontal diseases (69.6, n = 17) had the highest mean number of citations.

Characteristics of the included studies
The reviews included 2812 studies in which 246 were duplicated. Therefore, a total of 2565 studies were included in our analysis. Almost two-thirds of these studies (n = 1666) were published after 2000. The distribution of the location where the studies were conducted was quite heterogeneous. The USA, the UK, and India had the highest number of studies with 588 (22.9%), 221 (8.6%), and 164 (6.4%) studies, respectively. Sixty-eight studies were conducted using international collaborations. Figure 2 shows the distribution of the location of the studies.
Out of the 614 sources -reporting the RCTs, four were unpublished and one was a dissertation from the University of Sao Paulo, Brazil. However, 117 journals had more than three studies included in the Cochrane reviews. Most of these journals were published in the USA (47.9%) and the UK (26.4%).
The majority of the studies (n = 1600, 62.4%) had sample sizes of 50 or higher. The sample sizes' median was 303 (range:  and 86 studies had sample sizes larger than 1000. In contrast, 4.3% of the studies had less than 20 participants. The sample size was not reported in the tables in 17 studies. More than three-fourths of the included studies were parallel RCTs (n = 2040), followed by split-mouth (n = 253), cross-over (n = 154), and quasi-experimental (n = 72) studies. The design was unclear in one study. Mostly all the domains of the RoB tool were assessed in all the reviews. However, in 45 (1.8%) and 11 (0.4%) of the interventional studies, respectively, random sequence generation and allocation concealment were not assessed by the reviewers. Among the various bias domains, performance bias (blinding of the participants and personnel) showed the highest rate of high risk (31.4%). Instead, other sources of bias had the highest rate of low risk (68.2%). Figure 3 presents the overall assessment of each domain of the RoB tool. According to Fig. 4, random sequence generation data has continuously improved in terms of increasing low RoB during 2000-2019.
One hundred and forty-four studies had appeared in more than one Cochrane review with inconsistent RoB assessments in 107 (74.3%) of them. The assessment of attrition bias (incomplete outcome data; n = 207, 53.1%) and selection bias (allocation concealment; n = 34, 8.7%) had the highest and lowest rates of disagreement between the different reviews, respectively.

The Overall Risk of Bias (ORoB) and its association with the study characteristics
After removing the duplicate studies that had not reached consistent assessments concerning the RoBs (determined in the Cochrane reviews), 2475 studies remained for calculating the ORoB. Near half of these studies (n = 1119) had a high ORoB compared to 11.9% (n = 294) with a low ORoB. Of the 821 studies conducted before the year 2000, 6.6% had low ORoBs, compared to 14.5% of those conducted after 2000 and 15.5% of those conducted after 2010. The studies that used placebos had higher percentages of low ORoB (14.8% vs 10.7%) and lower percentages of high ORoB (27.6% vs 52.7%). The double-and triple-blind studies had higher percentages of low ORoB (23.6% and 23.3%, respectively). In comparison, the non-blinded studies had the lowest percentage of low ORoB (1.0%) and the highest percentage of high ORoB (80.4%). The studies with a cross-over design had the highest percentage of low ORoB (28.8%), followed by cluster RCTs (11.9%) and parallel studies (11.2%). The quasi-experimental mainly had a high ORoB (98.4%) ( Table 1). The high-ORoB studies had the highest mean sample size (n = 414.3), followed by the unclear (n = 213.6) and low (n = 190.0) ORoBs.

Discussion
Ranked as 23rd among the 37 subjects in terms of volume, the "dentistry and oral health" subject comprises 2.4% of all Cochrane reviews. Thus, there is still an evident paucity of dental evidence in both interventional and diagnostic aspects compared to other medical disciplines (20). Aside from the quantity, the research quality is paramount and needs to be continuously enhanced. The dearth of evidence in dentistry and oral health subjects intensi es the need for high-quality studies on which clinicians can rely. To evaluate the quality of dental clinical trials, the RoB and other characteristics of the studies included in the Cochrane oral health, systematic reviews were evaluated.
Approximately half of the clinical trials (47.3%) showed a high ORoB compared to 11.1% with a low ORoB. This nding was in line with the results reported by Yordanov et al., which revealed 43% of high ORoBs in medical clinical trials (21). Earlier assessments of RoB in dental clinical trials showed a signi cant decrease in studies judged as having inadequate methodological standards using the Cochrane risk of bias tool (11,17). Meanwhile, no speci c associations were identi ed to guide dental practitioners in optimising their future trials. The present ndings are highly suggestive of using control groups in oral health-related interventional studies as it could noticeably decrease the ORoB.
Furthermore, double-and triple-blind oral health-related interventional studies showed lower ORoBs compared to unblinded studies. Quasi-experimental studies showed higher ORoBs compared to crossover, cluster RCT, and parallel designs. These ndings can ultimately help clinical researchers direct their studies in more organised settings and designs and consider minimising probable RoBs in their relevant projects.
Even though dentists are able to conduct high-quality studies with low risks of performance bias (95.9% of the double-and triple-blinded studies), apparently, they have not been very eager to follow the blinding protocols of clinical interventional studies (cumulative percentage of double-and triple-blinded studies = 38.5%). This nding might be an implicit sign of incomprehensive dental research teams and the exclusive dispositions of clinical research in dentistry. Changing the exclusivity trend in research teams is proved to mainly impact the quality of evidence (22,23).
Despite knowing that using control groups decreases the ORoB, only 32% of the clinical trials reported an arm as a control group. Also, among the various RoB domains, performance bias (blinding of the participants and personnel) showed the highest rate of high-risk (31.4% of the studies). Consequently, setting correct methodological strategies and outlining the reporting protocol before conducting the study are imperative as well (21). These goals can be ensured by seeking clinical methodologists' professional help and consulting with statisticians when planning studies and outlining their stages (23). Providing better platforms for these communications and constantly promoting their feasibility features are critical (24). Another long-term solution for decreasing the RoBs is to enhance dental students and dental practitioners' knowledge about research methods (20,25,26). Teaching research methods should be pursued in parallel with the provision of more feasible and convenient ways for dental clinicians to contact professional methodologists, and a recommended step is to consult a statistician in every medical since they can help re ne the study design before the practical steps are taken. The patients are the ultimate stakeholders of these evidence-based decision-makings by dental practitioners since they can be provided with the most reliable and up-to-date therapeutic options (27)(28)(29).

Limitations
The Cochrane review authors made the judgements of RoB and some inaccuracies may be present in their perceptions and accordingly, their scorings. We did not assess the RoB for each clinical trial which could be considered both an advantage or a disadvantage. In doing so, we minimised the chances of misjudgement due to the massive number of studies. On the other hand, all the judgments of RoB assessments were done by the authors of Cochrane reviews, and we did not reassess these judgements. This could be deemed as a downside to our study. This study considers the fact that the blinding of the care providers is unachievable in some clinical trials in dentistry (e.g. surgical cases). Nonetheless, there are still various aspects of dental clinical trials that could be improved by simple, feasible actions, such as the accurate reporting of sequence generation and allocation strategies or using the most relevant yet feasible study designs.

Conclusion
Nearly half of the dentistry and oral health-related clinical trials (47.3%) showed high ORoBs.
Consequently, the quality of this dental evidence needs further enhancement and is inadequate in a noticeable part. If applicable, using methodological quality standards (e.g., blinding of the participants, using control groups, etc.) and using pre-determined study protocols (before conducting the research) and re ning study designs to the most randomised settings possible can decrease the ORoB.