Selective Outcome Reporting Bias Is Highly Prevalent in Randomized Clinical Trials of Periodontology

doi:10.21203/rs.3.rs-1562826/v1

PDF

Research Article

Selective Outcome Reporting Bias Is Highly Prevalent in Randomized Clinical Trials of Periodontology

https://doi.org/10.21203/rs.3.rs-1562826/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Objectives: This study aimed to investigate the prevalence of a type of bias named Selective Outcome Reporting (SOR) in publications of RCTs on non-surgical periodontal therapy (NSPT) and verify its associated factors.

Materials and Methods: The protocols were searched and selected on the www.clinicaltrials.gov platform up to January 16, 2022. Corresponding publications were identified, and data extraction and discrepancy analysis were performed. The risk of bias was assessed according to the RoB2 tool.

Results: One hundred forty-five studies (174 publications) were included. The prevalence of SOR was 49.7% and was unclear in nearly one third of studies (27.6%). Only 31.7% of the primary outcomes were completely described in the publications. The overall risk of bias was high in 60% of the included studies. SOR was associated with statistical significance (p < .001), and multiple publications of the same study (p = .005).

Conclusions: Our study demonstrated the high prevalence of SOR, highlighting the need to improve the quality of reporting of RCTs on NSPT studies.

Clinical relevance: SOR is a concerning bias that can affect clinical practice as it increases the effect of the intervention, emphasizing positive results. The incidence of SOR can be reduced by the prospective registry of the study protocol and the complete definition of the outcome. Journal editors could instruct reviewers to check discrepancies between the protocol and the manuscript whenever such information is available.

Bias

Selection Bias

Randomized Controlled Trials as Topic

Nonsurgical Periodontal Debridement

Periodontitis.

Evidence-based practice (EBP) consists of health care decision-making based on integrating clinical experience, patient values, and the best scientific evidence available [1, 2]. When it comes to decisions about clinical treatment, the randomized clinical trial (RCT) is the study that provides the highest level of scientific evidence about the efficacy of interventions [3]. However, the validity of the evidence can be threatened when biases are incorporated into the planning, conduct, and reporting of clinical trials [4, 5].

A specific form of bias that may be present in clinical trial publications is selective outcome reporting (SOR) [6]. SOR occurs when the primary outcome of a study protocol is changed or omitted, when a new outcome is added, or when the time point of the primary outcome is altered in the final publication [7, 8]. SOR directly affects EBP and may distort the effect of interventions, affecting clinical decision-making and public health policies [7, 8]. Furthermore, it may overestimate the effect of a treatment in the meta-analyses of systematic reviews [9, 10]. For these reasons, it is essential to identify the prevalence of SOR in RCTs in a given area of knowledge, and the factors that lead authors to introduce SOR in their publications.

The incomplete specification of outcomes can also compromise the transparency of clinical trial results [11, 12]. An outcome is completely specified when it is described in five levels: (a) domain, (b) specific measure, (c) specific metric, (d) aggregation method, and (e) time point [11, 13]. The incomplete specification of outcomes in the protocol or publication creates opportunities for authors to choose specific measures, metrics, and time points that were statistically significant [13].

Considering the important consequences of the occurrence of SOR, studies in the medical field have evaluated its prevalence with estimates ranging between 14% and 100% [14]. In dentistry, a previous study by our group identified SOR in 40.9% of the root coverage trials. Further, we observed an association of SOR with statistically significant results and the incomplete specification of outcomes [12].

Non-surgical periodontal therapy (NSPT) has been extensively studied for decades. Several therapies and adjuncts have been tested, and many outcomes have been used to measure the efficacy of these interventions [15]. However, it is necessary to investigate discrepancies between registered protocols and their respective publications regarding these outcomes.

So far, there is no information on the prevalence of SOR in clinical trials of NSPT. Therefore, this study aimed to assess outcomes discrepancies between protocols and publications in RCTs of NSPT and the variables associated to SOR.

Search strategy and eligibility criteria

We searched www.clinicaltrials.gov up to January 16, 2022 to identify RCT protocols in which one of the interventions consisted of NSPT. We used the keywords "periodontal" and "periodontitis" in the search and applied the filter "interventional (clinical trial)" to exclude observational studies at this stage of selection. We included the protocol if: a) the study was an RCT of patients diagnosed with periodontitis who have undergone NSPT with ultrasonic, manual instrumentation, or both, associated or not with adjunct therapies (e.g., laser, systemic antibiotics, local antimicrobials, etc.); b) at least one outcome in the registry was a clinical periodontal parameter such as probing depth (PD), clinical attachment level (CAL), or bleeding on probing (BOP), among others; c) at least one article related to the protocol was published in a peer-reviewed scientific journal. We excluded protocols in which NSPT was not evaluated, for example, studies of periodontal medicine, whose primary or secondary outcomes did not involve any periodontal parameter.

Selection of the protocols

Two reviewers (N.V.S. and A.C.N.) independently searched the protocols registered on the www.clinicaltrials.gov platform. When a protocol met the eligibility criteria, the same reviewers searched for publication references in the registry to identify any associated published articles. If publication references were unavailable in the registry, the reviewers searched the PubMed and Google Scholar databases, using the protocol identification provided by www.clinicaltrials.gov (NCT number), the principal investigator's name, and protocol-related keywords.

When two or more publications regarding the same protocol were identified, all data were collected and organized under the same NCT number. When no article returned from the search, the reviewers contacted the study's principal investigator by email. The study was not included when there were no responses after two contacts. All disagreements in study selection were solved by two experienced reviewers (J.C. and C.M.P.).

Data extraction

The data were extracted independently by the same reviewers (N.V.S. and A.C.N.) using a previously described extraction form [16] that addressed the protocol, the corresponding publication, and any discrepancies between them. Briefly, we extracted data about NCT number, principal investigator's name, country, and presence of industry funding for each protocol. In addition, we collected the protocol publication date and the start date of the study. To identify changes in the protocol after the initial record, a specific tool of www.clinicaltrials.gov called "history of changes" was used. We collected data regarding the original registration using this tool. According to the time of registration, the protocol was identified as prospectively or retrospectively registered. We considered that a study was prospectively registered if information about the registration was recorded before the inclusion of the first study subject. We also extracted information about the RCT design (parallel, split-mouth, factorial), blinding, number of arms, sample size, and follow-up period. The evaluation period of the results (time-points) for each protocol was collected. Data referring to primary and secondary outcomes were extracted according to protocol registration.

The following information of the corresponding publications was collected: the number of articles published related to the protocol, the name of the journal(s), their respective impact factor(s), and study start date (as published in the article). Furthermore, we verified if the authors reported the study registration, if the NCT number was available, and if any change in protocol was mentioned in the publication. Moreover, we extracted the number of arms and the tested interventions, follow-up period, industry funding, sample size, and sample size calculation. We also annotated the number of primary outcomes and if analysis of the primary and secondary outcome(s) was statistically significant (p < 0.05).

In cases where no outcome was identified as primary in the publication, we considered the outcome used to calculate the sample size as the primary. We considered the primary outcome unclear if it was not identified in the publication and if sample size calculation was not mentioned.

Analysis of discrepancies

We followed the criteria of Chan et al. [8], as modified by Mathieu et al. [7], to define SOR. Thus, we considered SOR to be present if a) an outcome recorded as primary in the protocol was reported as secondary in the publication; b) a secondary outcome in the protocol was reported as primary in the publication; c) a new primary outcome, which had not been described in the protocol, was introduced in the publication; d) a primary outcome of the protocol omitted in the publication; or e) there were discrepancies between protocol and publication(s) regarding primary outcome time-points.

We determined that SOR favored statistically significant results when: a) a secondary outcome in the protocol was upgraded to primary and reported as statistically significant in the publication, b) when a new primary outcome was reported as statistically significant in the publication, or c) when a primary outcome in the protocol was omitted or downgraded to secondary and reported as non-significant in the publication.

In addition, we assessed if there were other discrepancies regarding the study start date, presence or absence of industry funding, study design, number of arms, sample size, follow-up period, or presence of a new secondary outcome and its statistical significance.

Outcome specification levels

We performed an analysis of the complete outcome definition of each reported primary outcome to identify a possible association between SOR and the complete specification of the outcome. For this analysis, the primary outcomes were defined using five levels of specification: domain, specific measurement, specific metric, aggregation method, and time point [11, 13]. We also analyzed the specification in four levels, without the aggregation method, since in many protocols researchers only define the statistical analysis after verifying normality and homoscedasticity [11].

Risk of bias

We performed the risk of bias analysis to verify whether there was an association between SOR and bias related to study design, conduct, and reporting of results. The risk of bias was assessed in duplicate (N.V.S. and I.N.R.R.) according to the Risk of Bias 2 (RoB2) tool [17]. Disagreements were solved by two experienced reviewers (C.M.P. and J.C.).

In domain five, "bias in the selection of the reported result”, if SOR was found in our first analysis, we classified the study automatically with a high risk of bias for this domain. In addition, retrospectively recorded protocols were automatically judged to have "some concerns".

Statistical analyses

For the main analyses, the statistical unit was the study. Secondary analysis of the outcome complete description was performed, in which the statistical unit was the outcome. Regarding the study characteristics, frequency distributions for qualitative variables and means and standard deviations for quantitative variables were calculated. The presence or absence of SOR (dependent variable) was tested for possible association with the following independent variables: time of registration (prospective versus retrospective), industry funding (yes versus no), the significance of the primary study outcome (p ≤ .05 versus p > .05), number of publications resulting from the same protocol (one publication versus more than one publication), journal impact factor (smaller versus larger, using the median impact factor of the included journals as the cutoff point), risk of bias analysis (high versus low risk of bias or some concerns); complete definition of the primary outcome in the publication and in the protocol (studies in which the primary outcomes were defined entirely versus uncertain definition at one or more levels, except for the aggregation method). The chi-square test was used to test associations between independent variables and SOR. Statistical analyses were performed using the JAMOVI software (www.jamovi.org) and alpha was set at 5% for all tests.

Initially, we collected 1742 RCT protocols resulting from the search performed on www.clinicaltrials.gov on January 16, 2022 (Fig. 1). In the first screening phase, we excluded 1381 protocols after removing duplicates and protocols not meeting the eligibility criteria. We included 331 protocols in the second screening phase. Of these, we ultimately selected 145 protocols which resulted in 174 publications.

Characteristics of protocols and publications

The characteristics of the protocols are shown in Table 1. The majority of the studies were conducted in Asia (34.5%), Europe (28.3%), and South America (23.4%). According to the " history of changes " function, most protocols (78.6%) had no changes. Registration was performed retrospectively in 91.7% of cases, and only five studies (3.4%) reported the presence of industry funding. More than half of the studies had more than one primary outcome (55.2%), ranging between two and 16 outcomes. The study's primary outcome was unclear in one protocol (0.7%).

Table 1

Characteristics of the protocols of the included studies
Characteristics	Studies (n = 145)
Region of principal investigator - n (%) Asia Europe South America North America Africa	50 (34.5) 41 (28.3) 34 (23.4) 11 (7.6) 9 (6.2)
Protocol changes - n (%) Yes No	31 (21.4) 114 (78.6)
Timing of registration - n (%) Prospective Retrospective	12 (8.3) 133 (91.7)
Industry funding - n (%) Yes No	5 (3.4) 140 (96.6)
Study design as reported in the protocol - n (%) Paralell Split-mouth Crossover Factorial	124 (85.6) 11 (7.6) 5 (3.4) 5 (3.4)
Number of arms reported in the protocol - n (%) Min-max Two Three Four or more Unclear	2–10 104 (71.6) 30 (20.7) 10 (7.0) 1 (0.7)
Sample size - n Min-max Mean (SD)	11–816 72.0 (94.3.2)
Period of follow-up (months) Min-max Mean (SD)	48 hours – 60 months 7.45 (8.5)
Number of primary outcomes in the protocol - n (%) Min-max One More than one Unclear	1–16 64 (44.1) 80 (55.2) 1 (0.7)

Table 2 shows the characteristics of the 174 publications resulting from the 145 study protocols selected. Most studies (n = 132; 91%) resulted in a single publication. The remaining studies (n = 13; 9%) resulted in more than one publication, ranging from two to ten publications. The majority of the studies identified the protocol number in the publication (84.8%). In addition, no publication reported changes in the research protocol.

Many studies identified the primary outcome in the publication (81.4%). Approximately a third (32.4%) reported having only one primary outcome. In comparison, in 12.5% of studies, more than one primary outcome was reported, ranging from two to nine primary outcomes. The primary outcome was statistically significant in approximately half of the studies (n = 74; 51.1%).

Table 2: Characteristics of the publications of the included studies

Characteristics

Studies (n = 145)

Number of publications related to the same protocol - n (%)

One

Two

Three

Four

Ten

132(91.0)

7 (4.8)

2 (1.4)

3 (2.1)

1 (0.7)

NCT number identified in the publication - n (%)

Yes

123 (84.8)

22 (15.2)

Protocol change reported in publication - n (%)

Yes

0 (0.0)

145(100.0)

Industry-funded study reported in publication - n (%)

Yes

Unclear

11 (7.6)

127 (87.6)

7 (4.8)

Study design as reported in the publication - n (%)

Paralell

Split-mouth

Crossover

Factorial

Unclear

113 (77.9)

25 (17.2)

1 (0.7)

2 (1.4)

4 (2.8)

Number of arms reported in the publication – n (%)

Min-max

Two

Three

Four or more

2–8

106 (73.1)

29 (20.0)

10 (6.9)

Sample size calculation reported in the publication - n (%)

Yes

Unclear

118 (81.4)

21 (14.5)

6 (4.1)

Sample size - n

Min-max

Mean (SD)

5-823

67 (92.5)

Period of follow-up (months)

Min-max

Mean (SD)

1h-60 months

6.48 (7.75)

Number of primary outcomes in publication - n (%)

One

More than one

Unclear

47 (32.4)

18 (12.5)

80 (55.1)

Primary outcome identified in the publication - n (%)

Yes

118 (81.4)

27 (18.6)

Primary outcome statistically significant - n (%)

Yes

Unclear

74 (51.1)

45 (31.0)

26 (17.9)

Analysis of the complete definition of the outcome in protocols and publications

The distribution of primary outcomes in the protocols according to the three most frequent definition levels (domain, specific measure, specific metric, aggregation method, and time point) is shown in Online Resource 1.

We analyzed 250 primary outcomes. The most used domains were CAL (23.6%), PD (23.2%), and biochemical parameters (14.4%). In half of the cases (50.8%), the specific measure was not informed. However, when they were reported, the most frequent specific measures were: reduction in the depth of the bone defect (4.4%), C-reactive protein (CRP) (4%), and HbA1c (3.6%). Change from baseline was the most used specific metric, being reported in 58% of cases. The vast majority of protocols (91.2%) did not specify the aggregation method used. Assessment times varied widely between protocols, with 3-month and 6-month time points being the most common, occurring in 24.4% and 18% of cases, respectively.

Of the 250 outcomes presented, only 14 (5.6%) were completely defined at their five levels in the protocol (Online Resource 2). However, when we excluded the aggregation method level from the analysis, the proportion of protocols with a completely defined outcome was 29.6% (Online Resource 3). Despite this, the number of incompletely defined outcomes was predominant and occurred in 70.4% of the studies (Online Resource 3).

Online Resource 4 shows the analysis of the definition of the 186 primary outcomes in the publications. The most frequently used domain in publications was CAL (25.2%). More than a third of the publications (38.1%) did not specify the specific measure used, but when described, the most frequent specific measure was "CAL in all teeth," reported 21 times (11.3%). The most common specific metric was "change from baseline" (e.g., CAL gain, PD reduction, etc.), occurring in more than half of the publications (55.4%). However, this information was unclear in a high proportion of publications (27.9%). Mean was the most frequent aggregation method (65%). Most primary outcomes were presented with time points at three months (17.2%) or six months (8.6%), although this information was unavailable in 45.2% of cases. Figure 2 shows the two most used primary outcomes in publications of NSPT trials and their definitions.

Number of outcomes related to CAL and PD domains. The specification of the primary outcome was described in five levels (Mayo-Wilson et al. 2009), namely: o domain specific measure specific metric aggregation method time-point.

Online Resource 5 presents the definition of the primary outcome at five levels, as presented in the publication. Of the 186 primary outcomes analyzed, only approximately a third (31.7%) carried out the complete outcome definition, with all five levels well defined in the publication. When excluding the aggregation method from the analysis, complete outcome definition occurred in 35.5% (Online Resource 6).

Risk of bias analysis of included studies

We performed the risk of bias analysis in all included studies (n = 145) (Online Resource 7). Most of these studies (n = 87; 60%) had a high overall risk of bias (Fig. 3). Only one study (0.7%) was classified as having a low risk of bias, and in the remaining 39.3% of cases (n = 57), the studies had some concerns. Information on the risk of bias in each domain and their respective frequencies are presented in Online Resource 7.

Discrepancies in the primary outcome identified between the research protocol and the corresponding publication

We identified selective outcome reporting in 49.7% of publications (n = 72; Table 3). In 27.6% of the publications, SOR was unclear because the primary outcome was inadequately reported in the publication or the registry.

Table 3

Discrepancies identified between protocol and publication
Characteristics	Studies (n = 145)
Studies with Selective Outcome Reporting * Yes No Unclear	72 (49.7) 33 (22.7) 40 (27.6)
Primary outcome in the protocol described as secondary in the publication Yes No Unclear	36 (24.8) 77 (53.1) 32 (22.1)
Secondary outcome in the protocol described as primary in the publication Yes No Unclear	19 (13.1) 98 (67.6) 28 (19.3)
Protocol primary outcome not reported in publication Yes No Unclear	12 (8.3) 131 (90.3) 2 (1.4)
New primary outcome introduced in publication Yes No Unclear	16 (11.0) 111 (76.6) 18 (12.4)
Discrepancy in primary outcome time point Yes No Unclear	31 (21.4) 56 (38.6) 58 (40.0)
Selective outcome reporting favored statistical significance Yes No Unclear	56 (38.6) 48 (33.1) 41 (28.3)
New secondary outcome Yes No Unclear	87 (60.0) 57 (39.3) 1 (0.7)
New secondary outcome statistically significant in publication Yes No Unclear	71 (49.0) 72 (49.6) 2 (1.4)
New outcome Yes No	18 (12.4) 127 (87.6)
New outcome statistically significant in publication
Yes No Unclear	12 (8.3) 131(90.3) 2 (1.4)
All variables were described as n (%). * According to the criteria established by Chan et al. (2004), modified by Mathieu et al. (2009).

The discrepancies that led to SOR were: primary outcome of the protocol described as secondary in the publication (n = 36; 24.8%); secondary outcome in the protocol reported as primary in the publication (n = 19; 13.1%); omission of the protocol's primary outcome at publication (n = 12; 8.3%); a new primary outcome that was not reported in the protocol identified in the publication (n = 16; 11%); and discrepancies in assessment time-points of the primary outcome (n = 31; 21.4%). In addition, SOR clearly favored statistical significance in more than a third of the studies (n = 56; 38.6%), while this information was unclear in a further 28.3% of studies.

Other discrepancies between the study protocol and publication were in relation to: sample size (n = 66; 45.5%); study start date (n = 54; 37.2%); follow-up period (n = 25; 17.2%); study design (n = 14; 9.7%); number of arms (n = 10; 6.9%); and presence of industry funding (n = 7; 4.8%).

No statistically significant associations were found between SOR and risk of bias in the analyzed studies (in its four domains), industry funding, journal impact factor, study size, protocol changes, time of study registration, or with the incomplete definition of the primary outcome at both five and four levels (p > 0.05). However, SOR showed a significant association with statistical significance (p < 0.001), and more than one publication related to the same protocol (p < 0.005) (Table 4).

Table 4

Association between selective outcome reporting (SOR) and study characteristics
	SOR			Total (141)	p-value
Characteristics	Yes	No	Unclear
Timing of registration, per protocol Retrospective Prospective	67 (50.4) 5 (41.7)	29 (21.8) 4 (33.3)	37 (27.8) 3 (25.0)	133 (100.0) 12 (100.0)	0.656
Industry funding, per protocol Yes No	2 (40.) 70 (50.0)	2 (40.0) 31 (22.1)	1 (20.0) 39 (27.9)	5 (100.0) 140 (100.0)	0.644
Statistical significance in the publication Yes No Unclear	56 (100.0) 12 (25.0) 4 (9.8)	0 (0.0) 33 (68.8) 0 (0.0)	0 (0.0) 3 (6.2) 37 (90.2)	56 (100.0) 48 (100.0) 41(100.0)	< 0.001*
Journal impact factor ≥ 1 < 1	38 (56.7) 34 (43.6)	17 (25.4) 16 (20.5)	12 (17.9) 28 (35.9)	67 (100.0) 78 (100.0)	0.054
Protocol changes Yes No	21 (67.7) 51 (44.7)	6 (19.4) 27 (23.7)	4 (12.9) 36 (31.6)	31 (100.0) 114 (100.0)	0.053
Study Size Small Large	38 (52.1) 34 (47.2)	13 (17.8) 20(27.8)	22 (30.1) 18 (25.0)	73 (100.0) 72 (100.0)	0.350
Number of primary outcomes One More than one Unclear	28 (43.8) 44 (55.0) 0 (0.0)	21 (32.8) 12 (15.0) 0 (0.0)	15 (23.4) 24 (30.0) 1 (100.0)	64 (100.0) 80 (100.0) 1 (100.0)	0.060
Number of publications referring to the protocol One More than one	60 (45.5) 12 (92.3)	33 (25.0) 0 (0.0)	39 (29.5) 1 (7.7)	132 (100.0) 13 (100.0)	0.005*
Definition of the primary outcome(s) in the protocol Unclear definition of outcome(s) at one level or more Completely defined outcome(s) in 4 levels (aggregation method not included)	59 (56.2) 13 (32.5)	23 (21.0) 10 (27.5)	25 (22.8) 15 (40.0)	105 (100.0) 40 (100.0)	0.067
Definition of the primary outcome(s) in the publication Unclear definition of outcome(s) at one level or more Completely defined outcome(s) in 4 levels (aggregation method not included)	48 (49.5) 24 (50.0)	19 (19.6) 14 (29.2)	30 (30.9) 10 (20.8)	97 (100.0) 48(100.0)	0.289

All variables were described as n (%)

* Significant association according to the chi-square test.

We identified a high prevalence of SOR (n = 72; 49.7%) in NSPT trials. Further, our results showed an association of SOR with statistical significance of the primary outcome in the publication, and more than one publication resulting from the same study.

The high prevalence of SOR in the NSPT trials is similar to those observed in the medical field: 62% in the study by Chan et al. [8] and 49% both in surgical trials [18] and hematology publications [19]. In Dentistry, the prevalence was also high: 47% in the orthodontics field [20], 55.1% in dental implant studies [16], and 40.9% in root coverage trials [12]. Despite this high rate, the prevalence of SOR may have been underestimated for two reasons: 1) SOR was unclear in almost a third of the studies (n = 40; 27.6%) because authors did not specify the primary outcome in the publication; 2) we only used the www.clinicaltrials.gov platform to identify the protocols. Therefore, extending the search to other platforms could have increased SOR occurrence.

Most of the analyzed protocols (91.7%) were retrospectively recorded. This is a concerning result because late registration allows authors to change their outcomes once they have analyzed data. Retrospective registration creates room for publication bias and changes in the pre-specified primary outcome [21]. The rates of retrospective registration range from 46.9–79% in studies from the medical and dental fields [12, 16, 21, 22]. Study registration alone is not enough to increase transparency in reporting; it must be carried out prospectively before the inclusion of the first study participant [21, 23, 24].

Our study identified the primary outcome in 81.4% of the analyzed publications. A systematic review examining periodontal trials' primary outcomes between 2018 and 2020 showed that only half of the publications (54%) identified the primary outcome [25]. The low number of publications that identified the primary outcome is worrisome, since this oversight may increase the probability of authors selecting positive results, or omitting inconvenient results [26].

We analyzed the complete definition of the primary outcome both in the protocols and in their associated publications. According to the five specification levels, only 5.6% of the protocols had a completely specified outcome. A recent medical study also observed that the primary outcome was completely defined in only 3.3% of the trials, considering the five specification levels [27]. This percentage rose to 29.6% when we disregarded the aggregation method of the analysis. We carried out both analyses because in many studies, researchers only define the aggregation method in the statistical analysis after obtaining data and checking for normality and homoscedasticity [11]. On the other hand, we observed that almost a third (31.7%) of the publications presented the complete specification of the outcome when considering the five levels. After excluding the aggregation method from the analysis, this number increased to 35.5%. In a previous publication [12] we also observed a low rate (22.7%) of completely defined outcomes.

We did not identify an association between SOR and incomplete specification of the primary outcome in protocol and publication (p > 0.05). In contrast, Sendyk et al. [12] showed an association of SOR with an unclear definition of the primary outcome in the publication. Although no significant association was observed, it is worth noting observationally that there was SOR in 56.2% of the studies with an unclear definition of the outcome, as compared to 32.5% of the studies with a completely defined outcome. A possible explanation for this trend is that incomplete definition of the outcome can lead to cherry-picking, i.e., the investigator can perform multiple statistical analyses and select only the specific measures, metrics, or time points of the outcome that are statistically significant [13, 26]. These results emphasize the need to improve the description of the outcome, both in the protocol and in the publication.

Of the 145 included studies, 13 resulted in more than one publication referring to the same research protocol. Of these, 92.3% presented SOR. We observed SOR in the companion papers of these studies (i.e., from the second publication onwards). In these cases, the authors often 1) included a new primary outcome, 2) promoted the protocol's secondary outcome to the primary, or 3) omitted the protocol's primary outcome, recalculating the sample size for the outcome of interest. As a result, we observed a significant association between multiple publications and SOR. To prevent SOR and ensure transparency, authors must select a priori one primary outcome for the study and register this outcome in a publicly accessible registry. All resulting publications should mention the same original outcome as the primary [25], even in cases of companion papers that emphasize secondary outcomes.

SOR was associated with statistical significance, supporting previous findings [7, 8, 12, 19, 21, 28, 29]. The intention to select significant results to increase the chances of publication may be one of the reasons for the occurrence of SOR [8, 28, 30]. Furthermore, studies that interviewed authors to identify the reasons for the occurrence of SOR showed that the main justifications were a) lack of clinical importance, b) lack of statistical significance and c) space restrictions in the publication [8, 31, 32]. These findings corroborate Smyth et al. [33], who also demonstrated that there is misinformation by the authors, who do not understand the importance and consequences of not reporting all results in building the body of evidence. Omitting a non-significant finding or introducing a significant result can lead to overestimating the effectiveness of treatments and interfering with clinical practice [9, 10, 28, 34].

One of the limitations of this investigation is that the protocol search strategy was restricted to the www.clinicaltrials.gov platform, which may have compromised the external validity of our research. On the other hand, other registration platforms do not have the "history of changes" tool that identifies any changes made to the research protocol after its registration. Therefore, we chose to use www.clinicaltrials.gov because of the possibility of analyzing these changes.

Some measures can improve reporting transparency and reduce the incidence of SOR, such as 1) prospective registration of the study before the inclusion of the first research patient; 2) the complete definition of the primary outcome both in the research protocol and in the final publication; 3) changes in the planning and conduct of the ECR informed appropriately and transparently. In addition, journal editors should instruct reviewers to check for discrepancies between the protocol and the manuscript whenever such information is available. These efforts can increase the transparency of clinical trials, strengthening evidence-based practice in Periodontology and other fields of dentistry.

We observed a high prevalence of SOR in RCTs concerning non-surgical periodontal therapy. SOR was associated with statistical significance, and the number of publications referring to the same study.

Authors contributions

N.V.S.: Contributed to conception, design, data acquisition and interpretation, drafted and critically revised the manuscript. A.C.N. and I.N.R.R. contributed to data acquisition and interpretation, and critically revised the manuscript. D.I.S. and J.C. contributed to conception, design, and critically revised the manuscript. C.M.P.: Contributed to conception, design, performed all statistical analyses, drafted, and critically revised the manuscript. All authors gave final approval and agreed to be accountable for all aspects of the work.

Funding:

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001; and the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq).

Ethical approval and informed consent. No approval and consent were needed.

Conflict of Interest: The authors declare that they have no conflict of interest.

Sackett DL, Rosenberg WM, Gray JA, et al (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312:71–2
Chiappelli F (2019) Evidence-Based Dentistry: Two Decades and Beyond. J Evid Based Dent Pract 19:7–16. https://doi.org/10.1016/j.jebdp.2018.05.001
Howick J, Chalmers I, Glasziou P, et al (2011) The Oxford 2011 Levels of Evidence. http://www.cebm.net/index.aspx?o=5653. Accessed 22 Mar 2021
Hinton S, Beyari MM, Madden K, Lamfon HA (2015) The Risk of Bias in Randomized Trials in General Dentistry Journals. J Long Term Eff Med Implants 25:277–288. https://doi.org/10.1615/jlongtermeffmedimplants.2015011621
Papageorgiou S, Kloukos D, Petridis H, Pandis N (2015) An Assessment of the Risk of Bias in Randomized Controlled Trial Reports Published in Prosthodontic and Implant Dentistry Journals. Int J Prosthodont 28:586–593. https://doi.org/10.11607/ijp.4357
Boutron I, Page M, Higgins J, et al (2020) Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ WV (ed) Cochrane Handbook for Systematic Reviews of Interventions version 6.1. Available from .
Mathieu S, Boutron I, Moher D, et al (2009) Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials. JAMA 302:977. https://doi.org/10.1001/jama.2009.1242
Chan AW, Hróbjartsson A, Haahr MT, et al (2004) Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials. JAMA 291:2457. https://doi.org/10.1001/jama.291.20.2457
Kirkham JJ, Dwan KM, Altman DG, et al (2010) The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340:637–640. https://doi.org/10.1136/bmj.c365
Kirkham JJ, Altman DG, Chan A, et al (2018) Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews to include all relevant studies. BMJ 362:k3802:1–5. https://doi.org/10.1136/bmj.k3802
Zarin DA, Tse T, Williams RJ, et al (2011) The ClinicalTrials.gov results database - Update and key issues. N Engl J Med 364:852–860. https://doi.org/10.1056/NEJMsa1012065
Sendyk DI, Souza NV, César Neto JB, et al (2021) Selective outcome reporting in root coverage randomized clinical trials. J Clin Periodontol 48:867–877. https://doi.org/10.1111/jcpe.13451
Mayo-Wilson E, Fusco N, Li T, et al (2017) Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol 86:39–50. https://doi.org/10.1016/j.jclinepi.2017.05.007
Li G, Abbade LPF, Nwosu I, et al (2018) A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research. BMC Med Res Methodol 18:. https://doi.org/10.1186/s12874-017-0465-7
Pannuti CM, Costa FO, Souza NV, et al (2021) Randomized clinical trials in periodontology: focus on outcomes selection. Braz Oral Res 35:. https://doi.org/10.1590/1807-3107bor-2021.vol35.0100
Sendyk DI, Rovai ES, Souza NV, et al (2019) Selective outcome reporting in randomized clinical trials of dental implants. J Clin Periodontol 46:jcpe.13128. https://doi.org/10.1111/jcpe.13128
Sterne JAC, Savović J, Page MJ, et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ l4898. https://doi.org/10.1136/bmj.l4898
Hannink G, Gooszen HG, Rovers MM (2013) Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg 257:818–823. https://doi.org/10.1097/SLA.0b013e3182864fa3
Wayant C, Scheckel C, Hicks C, et al (2017) Evidence of selective reporting bias in hematology journals: A systematic review. PLoS One 12:1–13. https://doi.org/10.1371/journal.pone.0178379
Koufatzidou M, Koletsi D, Fleming PS, et al (2019) Outcome reporting discrepancies between trial entries and published final reports of orthodontic randomized controlled trials. Eur J Orthod 41:225–230. https://doi.org/10.1093/ejo/cjy046
Jones CW, Platts-Mills TF (2012) Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med 60:458–464.e1. https://doi.org/10.1016/j.annemergmed.2012.02.005
Farquhar CM, Showell MG, Showell EAE, et al (2017) Clinical trial registration was not an indicator for low risk of bias. J Clin Epidemiol 84:47–53. https://doi.org/10.1016/j.jclinepi.2016.11.011
Zarin DA, Keselman A (2007) Registering a clinical trial in ClinicalTrial.gov. Chest 131:909–912. https://doi.org/10.1378/chest.06-2450
Pannuti CM, Sendyk DI, Graças YT das, et al (2020) Clinically relevant outcomes in dental clinical trials: challenges and proposals. Braz Oral Res 34:. https://doi.org/10.1590/1807-3107bor-2020.vol34.0073
Al-Shamsi M, Mehta J, Nibali L (2021) Study design and primary outcome in randomized controlled trials in periodontology. A systematic review. J Clin Periodontol 1–8. https://doi.org/10.1111/jcpe.13443
Andrade C (2015) The primary outcome measure and its importance in clinical trials. J Clin Psychiatry 76:e1320–e1323. https://doi.org/10.4088/JCP.15f10377
Vrljičak Davidović N, Komić L, Mešin I, et al (2021) Registry versus publication: discrepancy of primary outcomes and possible outcome reporting bias in child and adolescent mental health. Eur Child Adolesc Psychiatry. https://doi.org/10.1007/s00787-020-01710-5
Zhang S, Liang F, Li W (2017) Comparison Between Publicly Accessible Publications, Registries, and Protocols of Phase III Trials Indicated Persistence of Selective Outcome Reporting. J Clin Epidemiol 87–94. https://doi.org/10.1016/j.jclinepi.2017.07.010.This
Aggarwal R, Oremus M (2019) Selective outcome reporting is present in randomized controlled trials in lung cancer immunotherapies. J. Clin. Epidemiol.
Dwan K, Gamble C, Williamson PR, Kirkham JJ (2013) Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias - An Updated Review. PLoS One 8:. https://doi.org/10.1371/journal.pone.0066844
Chan AW (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Can Med Assoc J 171:735–740. https://doi.org/10.1503/cmaj.1041086
Chan AW, Altman DG (2005) Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 330:753. https://doi.org/10.1136/bmj.38356.424606.8F
Smyth RMD, Kirkham JJ, Jacoby A, et al (2011) Frequency and reasons for outcome reporting bias in clinical trials: Interviews with trialists. Bmj 342:155. https://doi.org/10.1136/bmj.c7153
Wayant C, Aran G, Johnson BS, Vassar M (2020) Evaluation of Selective Outcome Reporting Bias in Efficacy Endpoints in Print and Television Advertisements for Oncology Drugs. J Gen Intern Med 35:2853–2857. https://doi.org/10.1007/s11606-020-06028-1

No competing interests reported.

SupplementaryTablesSORNSPTCOI2022.04.12.docx

PDF

Editor assigned by journal
23 Apr, 2022
Submission checks completed at journal
21 Apr, 2022
First submitted to journal
15 Apr, 2022

You are reading this latest preprint version

Selective Outcome Reporting Bias Is Highly Prevalent in Randomized Clinical Trials of Periodontology

Status:

Version 1

Abstract

Figures

Introduction

Material And Methods

Search strategy and eligibility criteria

Selection of the protocols

Data extraction

Analysis of discrepancies

Outcome specification levels

Risk of bias

Statistical analyses

Results

Characteristics of protocols and publications

Analysis of the complete definition of the outcome in protocols and publications

Risk of bias analysis of included studies

Discrepancies in the primary outcome identified between the research protocol and the corresponding publication

Discussion

Conclusion

Declarations

References

Competing Interests

Supplementary Files

Status:

Version 1