Spin on adverse effects in abstracts of systematic reviews of orthodontic interventions: A cross-sectional study (Part 2)

doi:10.21203/rs.3.rs-1349917/v1

Download PDF

Research Article

Spin on adverse effects in abstracts of systematic reviews of orthodontic interventions: A cross-sectional study (Part 2)

https://doi.org/10.21203/rs.3.rs-1349917/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 20 Jun, 2023

Read the published version in Systematic Reviews →

You are reading this latest preprint version

Background. It is critical that abstracts of systematic reviews transparently report both the beneficial and adverse effects of interventions without misleading the readers. This cross-sectional study assessed whether adverse effects of interventions were reported or considered in abstracts of systematic reviews of orthodontic interventions and whether spin on adverse effects was identified when comparing the abstracts with what was sought and reported in these reviews.

Methods. This cross-sectional study (part 2 of 2) used the same sample of 98 systematic reviews orthodontic interventions on human patients that was used in part 1. Eligible reviews were retrieved from the Cochrane Database of Systematic Reviews and the 5 leading orthodontic journals between August 1 2009 and July 31 2021. Prevalence proportions were sought for 3 outcomes as defined in the published protocol. Univariable logistic regression models were built to explore associations between the presence of spin in the abstract and predictors such as the journal, year, number of authors, conflict of interest reported, funding, and type of orthodontic intervention. Odd ratios (OR) 95% confidence intervals (95% CI) were used to quantify respectively the strength of associations and their precision.

Results. The majority, 76.5% (75/98), of eligible reviews reported or considered (i.e., discussed, weighted etc.), potential adverse effects of orthodontic interventions in the abstract and the proportion of spin on adverse effects was 40.8% (40/98) in the abstract of these reviews. Misleading reporting was the predominant category of spin, i.e., 90% (36/40). The explorative analyses did not find an association between the presence of spin in the abstract and any of the predictors.

Conclusion. End users of systematic reviews of orthodontic interventions have to be careful when interpreting the results on adverse effects in the abstracts of these reviews, because they could be jeopardized by uncertainties such as not being reported and misleading reporting as a result of spin. Editors, peer reviewers, researchers, and guideline developers have key responsibilities in tackling these problems. This is particularly important, because titles and abstracts are often the only read sections of papers in biomedicine.

Orthodontics

Reporting

Systematic review

Intervention

Spin

Misleading reporting

Misleading interpretation

Misleading extrapolation

Adverse effect

Adverse event

Harm

Safety

Abstracts should provide key information on a research study, which helps readers decide whether or not to access the full report [1]. It is therefore critical that abstracts transparently report the results of both the beneficial and adverse effects of interventions without misleading the readers. Misleading reporting, misleading interpretation, and misleading extrapolation of study results has been called ‘spin’ [2, 3]. In this study we assessed whether adverse effects of interventions were reported or considered in abstracts of both Cochrane and non-Cochrane reviews of orthodontic interventions and whether spin and what type of spin regarding adverse effects was present when comparing the abstracts with what was sought and reported in these reviews.

Titles and abstracts of publications of healthcare interventions are used for multiple purposes such as: (1) an initial screening of the study type (2) clarifying the included type of patients, interventions, comparators, outcomes, and settings (3) obtaining a summary of the findings and (4) an initial assessment of the validity of the study [1, 4, 5]. Titles and abstracts are the most and often only read sections of biomedical papers, because of a lack of time of readers, paywalls or language issues [6]. It is therefore important that abstracts can be used as stand-alone documents that clearly and truthfully reflect what was reported in the full text [7]. The standard for Methodological Expectations of Cochrane Intervention Reviews (MECIR) [8] states under Item R13 that: ‘The abstract of the review should aim to reflect a balanced summary of the benefits and harms of the intervention’ and this is a ‘mandatory’ Cochrane review standard. The inclusion of ‘adverse effects’ in this standard is crucial, because these effects are often poorly assessed and reported in clinical trials and systematic reviews of healthcare interventions [9–16].

In this context it is important that findings on adverse effects are presented accurately in the abstract without misleading the reader. ‘A distorted presentation of study results’ has been called ‘spin’ [2, 3]. This definition and other commonly used definitions of spin and key terminology used in this article are listed in Table 1 [2,3,17–25]. Spin has been subdivided in 3 categories: ‘misleading reporting’, ‘misleading interpretations’, and ‘misleading extrapolations’ of study results’ [2]. We adopted the definitions by Lazarus et al. [23] for these 3 categories of spin (Table 1). Controlling spin is important, because of its high prevalence and its consequences. For example, a randomized controlled trial (RCT) showed that spin in abstracts can influence the clinician’s interpretation of the results of a study [26]. Further, Yavchitz et al. [27] showed that the presence of spin in press releases and the mass media was related with spin in the conclusions of the pertinent abstracts of peer-reviewed RCTs. Our scoping searches showed that a high prevalence of spin has been recorded in abstracts of numerous research studies and for a wide variety of disciplines. For example spin was present in: 84% (107/128) of abstracts of reports of non-randomized studies assessing an intervention [23]; 23% (24/105) of abstracts of RCTs in rheumatology [28]; 57% (53/93) of abstracts of cardiovascular RCT reports [29]; 34.2% (25/73) of abstracts of systematic reviews and meta-analyses related to treatment of proximal humerus fractures [30]; 37.6% (27/72) of results and 58.3% (42/72) of conclusions of abstracts of parallel-group RCTs with statistically non-significant results (P ≥ .05) [25]. Spin in abstracts of orthodontic studies was assessed in 2 recent publications [31,32]. Guo et al. [31] found spin in 62.2% (69/111) of abstracts of parallel-group RCTs with clearly stated statistically non-significant primary outcomes and Makou et al. [32] identified spin in 48.6% (53/109) of abstracts of orthodontic meta-analyses.

Table 1

Glossary of terms
Term	Definition
Systematic review	Cochrane [17] defines a systematic review as follows: ‘A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.’
Intervention review	Cochrane [17] defines an intervention review as follows: ‘Intervention reviews assess the benefits and harms of interventions used in healthcare and health policy.’
Orthodontic interventions	Steegmans et al. [18] define orthodontic interventions as follows: ‘Orthodontic interventions refer to the use of any type of orthodontic appliance to move teeth or change the jaw size or position for orthodontic purposes. These interventions also include appliances to maintain or stabilize the results of orthodontic treatment, for example retainers.’
Adverse effect	Cochrane [19,20], defines an adverse effect as ‘an adverse event for which the causal relation between the intervention and the event is at least a reasonable possibility’.
Spin [3]	‘Distorted presentation of study results’.
Spin [3]	‘A misrepresentation of study results, regardless of motive (intentionally or unintentionally) that overemphasizes the beneficial effects of the intervention and overstates safety compared with that shown by the results’.
Spin [21]	‘A specific intentional or unintentional reporting that fails to faithfully reflect the nature and range of findings and that could affect the impression the results produce in readers’
Spin [22]	‘A specific reporting that fails to faithfully reflect the nature and range of findings and that could affect the impression that the results produce in readers, a way to distort science reporting without actually lying’
Misleading reporting related spin [23]	‘Incomplete reporting of the study results that could be misleading for the reader’.
Misleading interpretation related spin [23]	Inadequate interpretation of the study results overestimating the beneficial effect of the intervention.
Misleading (inappropriate) extrapolation related spin [23]	‘Inappropriate generalization of the study results by inadequate 1) extrapolation from the population, interventions or outcome actually assessed in the study to a larger population, different interventions or outcomes, or 2) inadequate implications for clinical practice.
Spin (in the abstract) on adverse effects of interventions [24]	Incomplete or inadequate reporting, interpretation, or extrapolation (or a combination of these variables) of findings on adverse effects of interventions in the abstract that could be misleading for the reader.

This is part 2 of 2 cross-sectional studies on assessing and reporting of adverse effects in systematic reviews of orthodontic interventions published in 5 leading orthodontic journals and in the Cochrane Database of Systematic Reviews. Part 1 focused predominantly on seeking and reporting of adverse effects in the main text and supplementary files of these reviews [18,33]. In part 2 we assessed whether adverse effects of orthodontic interventions were reported or considered (i.e., discussed, weighed etc.) in abstracts of these reviews. We further measured whether spin was introduced in the abstract regarding information on adverse effects as found and reported in these reviews. We also assessed the different categories of spin. The findings of this research study are important for patients and clinicians, but also for researchers, peer reviewers and editors because they have a crucial role in reducing the prevalence of spin [34].

Objectives

Our objectives were presented in the following 3 research questions [24]. Recent (up to October 31 2021) scoping searches showed that these questions were not assessed previously.

Question 1. In abstracts of systematic reviews of orthodontic interventions were potential adverse effects of these interventions reported or considered (i.e., discussed, weighed etc.)?
Question 2. Was spin identified on adverse effects of orthodontic interventions in the abstract?
Question 3. What type of spin was identified on adverse effects of orthodontic interventions in the abstract?

This manuscript reports the methods and results of part 2 of a cross-sectional study using the same 98 eligible reviews as in part 1 [33]. To avoid redundancy some sections of this manuscript have been abridged, but all additional information on the research methods and the characteristics of the included reviews can be found in part 1 [33] and in the published protocols of parts 1 and 2 [18,24]. The protocol for this second cross-sectional study [24] can be consulted via the following link: https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-019-0084-4

The checklist of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for cross-sectional studies [35] was included as additional file 1. The differences between the methods planned in our protocol and those implemented in the final study were reported in Additional file 2. The rationales for these differences were also given. All raw data were reported in the Open Science framework Open Science Framework (https://osf.io/ka7mp/). All eligible studies for this cross-sectional study are reported in Additional file 3. There was no patient or public involvement during the development of the protocol or in the conduct of this study.

Eligibility criteria

The eligibility criteria were published previously in our protocol [24] and in part 1 of this study [33] and are given in additional file 2 [36].

Information sources and search strategy

The Cochrane Database of Systematic reviews [37] and the websites of 5 leading orthodontic journals were the information sources of this study. These 5 orthodontic journals are: European Journal of Orthodontics [EJO], American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], Angle Orthodontist (AO), The Korean Journal of Orthodontics (KJO), and Orthodontics and Craniofacial Research (OCR). These journals were manually searched from August 1 2009 until July 31 2021 for systematic reviews that fulfilled the eligibility criteria.

Study records

Data collection procedures

Two calibrated authors (PS and RMR) independently collected data from the 98 eligible reviews to address the research questions. In the case of disagreement on these procedures the following strategies were implemented and in this sequence: (1) discussions between these operators (2) contacting of authors by email to clarify issues regarding a specific manuscript (3) consultation with a methodologist (NDG) to arbitrate. Data extraction was conducted according to our published protocol [24] and all data collection forms were reported in (Additional file 2). Data items were collected from the entire eligible review, i.e., the entire manuscript including the abstract, tables, figures, and additional files. We implemented this procedure for all eligible reviews, but did not extract data from the plain language summary of eligible Cochrane reviews.

Assessing adverse effects of orthodontic interventions

Pain and the various categories of adverse effects hypothetically linked to orthodontic interventions as defined by Preoteasa et al. [38] and modified by Steegmans et al. [33] were reported in a table in Additional file 2. This table was used as our reference to assess the reporting on adverse effects in the abstract. When additional adverse effects were identified that were not given in this table we included them with rationale. Effects that could be labelled either as ‘beneficial’ or ‘adverse’ were not included unless the review authors labelled these ambiguous effects as ‘adverse’. Explanations for such decisions were given.

Assigning spin of adverse effects of orthodontic interventions in abstracts of systematic reviews

Spin was assigned by comparing whether what was reported in the abstract on adverse effects of orthodontic interventions was congruent with the findings on these effects in the review. Three types of spin were assigned i.e., misleading reporting, misleading interpretation, and misleading (inappropriate) extrapolation on adverse effects of orthodontic interventions in abstracts of systematic reviews [27]. To facilitate this assignment each type of spin was subdivided in categories and spin was assigned when it was identified in one or more of these categories. Spin was assessed in all eligible reviews irrespective of whether these reviews sought adverse effects of interventions or not. Because the pilot tests for our protocol identified only 2 reviews with spin [24] and because assessing spin is not easy [22], we conducted additional pilot tests on 10 RCTs to further calibrate the operators (PS and RMR) that assigned spin and to fine-tune the descriptions of spin and the checklists for assigning spin. These fine-tuned descriptions of spin and the pertinent data collection forms to identify spin are reported respectively in Table 2 and Additional file 2.

Table 2

Types of spin in reviews that did or did not seek adverse effects of interventions
Definitions of the 3 types of spin	Reviews that sought adverse effects of interventions	Reviews that did not seek adverse effects of interventions
Misleading reporting (in the abstract) on adverse effects of interventions: ‘Incomplete or inadequate reporting in the abstract on the results of adverse effects that were not supported by the findings of the review’.	Categories: 1) Not reporting in the abstract on the results of adverse effects found in the review. 2) Selective reporting in the abstract on the results of adverse effects found in the review.	Categories: 1) Reporting on results of adverse effects in the abstract when adverse effects were not sought. 2) Reporting in the abstract that adverse effects were sought when they were not sought.
Misleading interpretation (in the abstract) on adverse effects of interventions: ‘Interpretation in the abstract on the results of adverse effects that was not supported by the findings of the review’. This could for example underestimated the adverse effects of the intervention’.	Categories: 1) Claiming in the abstract that the intervention is safe (has no or minimal adverse effects), despite concerning results on adverse effects found in the review e.g., based on non-statistically significant results on adverse effects with wide confidence intervals [27]. 2) Downgrading in the abstract the importance of the adverse effects, despite concerning results on adverse effects found in the review. 3) Recommendations are made in the abstract for clinical practice that are not supported by the findings in the review on adverse effects’[27].	Categories: 1) Claiming in the abstract that the intervention is safe (has no or minimal adverse effects) despite not having sought adverse effects. 2) Downgrading in the abstract the importance of the adverse effects, despite not having sought adverse effects. 3) Recommendations are made in the abstract for clinical practice despite not having sought adverse effects.
Misleading (inappropriate) extrapolation (in the abstract) on adverse effects of interventions: ‘Overgeneralisation in the abstract of the study results to different populations, interventions, outcomes or settings than were assessed in the review despite evidence on adverse effects on a different population, intervention, outcome or setting’.	Categories: 1) Results are extrapolated in the abstract to another population, intervention, outcome or setting than were assessed in the review despite evidence on adverse effects on a different population, intervention, outcome or setting.	Categories: 1) Results are extrapolated in the abstract to another population, intervention, outcome or setting than were assessed in the review despite not having sought adverse effects.

Power calculation

In our pilot sample we identified an overall proportion of 14.3% (2/14) of spin of the adverse effects in the abstracts of 14 systematic reviews of orthodontic interventions. For this proportion Epitools software [39] calculated a required sample size of 48 studies (precision 0.1 and confidence level 0.95), which was fulfilled by our 98 eligible reviews.

Outcomes and statistical analyses

Outcomes

Prevalence proportions were calculated to quantify the answers to our 3 research questions in the 98 selected reviews. According to our published protocol these proportions were also calculated separately for reviews that either did (n = 84) or did not (n = 14) seek any findings related to adverse effects of interventions in the included studies [33]. These statistics were calculated for (1) all journals as a one group (2) the five leading orthodontic journals together and the Cochrane Database of Systematic Reviews separately and (3) each eligible journal separately.

Explorative analyses

Univariable logistic regression models were built to determine the association between the presence of spin in the abstract and characteristics of the systematic review i.e., journal, year of publication, number of authors, conflict of interest reported, conflict of interest present, funding reported, and type of orthodontic intervention. These analyses were not registered in the protocol and should therefore be interpreted as exploratory. The strengths of associations were quantified using odds ratios (OR), and 95% confidence intervals (95% CI). Multivariable models were built if multiple significant predictors were found in the univariable analysis. Analyses were performed with the use of commercial software (IBM SPSS 22.0, SPSS Inc, Chicago, IL). A two-sided P value of 0.05 was considered to be statistically significant.

The results of the search for eligible reviews were reported previously [33] and identified 98 eligible reviews. All included reviews are reported in Additional file 3. Figures 1 and 2 present flow diagrams of the answers to the research questions and Tables 3 and 4 report the pertinent proportion statistics. The number of identified systematic reviews and the number of eligible systematic reviews of orthodontic interventions given in these tables were published previously [33] and were reported again to give context to the outcomes to our research questions. In these tables outcomes are further subdivided for eligible reviews that did (n = 84) or did not (n = 14) seek any findings related to adverse effects of interventions in the included studies. The initial inter-operator agreement between both operators for assigning spin was high (Cohen’s κ = 0.94) and complete agreement between operators was reached after discussion.

Table 3

Outcomes on reporting or considering adverse effects and spin in abstracts of systematic reviews of orthodontic interventions
Description of outcomes	Cochrane	EJO	AJODO	AO	KJO	O&CR	All orthodontic journals	All journals
The number of identified systematic reviews	n = 29	n = 100	n = 68	n = 61	n = 11	n = 53	n = 293	n = 322
The number of eligible systematic reviews of orthodontic interventions	n = 10	n = 28	n = 20	n = 23	n = 3	n = 14	n = 88	n = 98
The prevalence of eligible systematic reviews of orthodontic interventions that sought any findings related to adverse effects of interventions in the included studies*	100.0% (10/10)	92.9% (26/28)	75.0% (15/20)	78.3% (18/23)	100.0% (3/3)	85.7% (12/14)	84.1% (74/88)	85.7% (84/98)
Outcome 1: The prevalence of eligible systematic reviews in which potential adverse effects of interventions were reported or considered (i.e. discussed, weighted etc.) in the abstract*	100.0% (10/10)	82.1% (23/28)	65.0% (13/20)	65.2% (15/23)	100.0% (3/3)	78.6% (11/14)	73.9% (65/88)	76.5% (75/98)
Outcome 1a: The prevalence of eligible systematic reviews in which potential adverse effects of interventions were reported or considered (i.e. discussed, weighted etc.) in the abstract*	100% (10/10)	88.5% (23/26)	86.7% 13/15)	77.8% (14/18)	100% (3/3)	91.7% (11/12)	86.5% (64/74)	88.1% (74/84)
Outcome 1b: The prevalence of eligible systematic reviews in which potential adverse effects of interventions were reported or considered (i.e. discussed, weighted etc.) in the abstract*	00.0% (0/0)	0.0% (0/2)	0.0% (0/5)	20.0% (1/5)	0.0% (0/0)	0.0% (0/2)	7.1% (1/14)	7.1% (1/14)
Outcome 2: The prevalence of eligible systematic reviews in which spin was identified on adverse effects of orthodontic interventions in the abstract*	60.0% (6/10)	35.7% (10/28)	30.0% (6/20)	39.1% (9/23)	66.7% (2/3)	50.0% (7/14)	38.6% (34/88)	40.8% 40/98
Outcome 2a: The prevalence of eligible systematic reviews in which spin was identified on adverse effects of orthodontic interventions in the abstract*	60.0% (6/10)	38.5% (10/26)	40.0% (6/15)	44.4% (8/18)	66.7% (2/3)	58.3% (7/12)	44.6% (33/74)	46.4% (39/84)
Outcome 2b: The prevalence of eligible systematic reviews in which spin was identified on adverse effects of orthodontic interventions in the abstract*	0.0% (0/0)	0.0% (0/2)	0.0% (0/5)	20.0% (1/5)	0.0% (0/0)	0.0% (0/2)	7.1% (1/14)	7.1% (1/14)
*Outcome 1a addresses research question 1a, outcome 1b addresses research question 1b, and outcome 1 addresses the answers to questions 1a and 2b combined, outcome 2a addresses research question 2a, outcome 2b addresses research question 2b, and outcome 2 addresses the answers to questions 2a and 2b combined. (See Figs. 1 and 2).

Table 4

Outcome 3: Types and categories of spin in abstracts of systematic reviews of orthodontic interventions
Type of spin and category	Cochrane	EJO	AJODO	AO	KJO	O&CR	All orthodontic journals	All journals
Misleading reporting Category: Not reporting in the abstract on the results of adverse effects found in the review.	n = 2	n = 3	n = 2	n = 5	n = 0	n = 1	n = 11 Prevalence: 32.4% (11/34)	n = 13 Prevalence: 32.5% (13/40)
Misleading reporting Category: Selective reporting in the abstract on the results of adverse effects found in the review.	n = 4	n = 6	n = 4	n = 2	n = 2	n = 5	n = 19 Prevalence: 55.9% (19/34)	n = 23 Prevalence: 57.5% (23/40)
Misleading interpretation Category: Claiming in the abstract that the intervention is safe (has no or minimal adverse effects), despite concerning results on adverse effects found in the review		n = 1		n = 1		n = 1	n = 3 Prevalence: 8.8% (3/34)	n = 3 Prevalence: 7.5% (3/40)
Misleading (inappropriate) extrapolation Category: Results are extrapolated in the abstract to another population, intervention, outcome or setting than were assessed in the review despite evidence on adverse effects on a different population, intervention, outcome or setting.’				n = 1			n = 1 Prevalence: 2.9% (1/34)	n = 1 Prevalence: 2.5% (1/40)

The majority 76.5% (75/98) of eligible reviews reported or considered (i.e., discussed, weighted etc.) potential adverse effects of orthodontic interventions in the abstract and the total proportion of spin on adverse effects in the abstract was 40.8% (40/98) in these reviews (Table 3). Misleading reporting was the predominant category of spin, i.e., 90% (36/40) (Table 4). Our explorative analyses did not find an association between the presence of spin in the abstract and any of the predictors, i.e., journal, year of publication, number of authors, conflict of interest reported, conflict of interest present, funding reported, and type of orthodontic intervention (Table 5).

Table 5

Associations between presence of spin in the abstract and characteristics of the systematic review
Item	Variable insertion in the model	Description	Yes (%)	No (%)	OR	Lower 95%CI	Upper 95%CI	P value
Journal	Categorical	Cochrane	4 (40.0%)	6 (60.0%)	1	-	-	-
		EJO	18 (64.3%)	10 (35.7%)	0.37	0.08	1.63	0.19
		AJODO	14 (70.0%)	6 (30.0%)	0.29	0.06	1.39	0.12
		AO	14 (60.9%)	9 (39.1%)	0.43	0.09	1.95	0.27
		KJO	1 (33.3%)	2 (66.7%)	1.33	0.09	20.11	0.84
		O&C	7 (50.0%)	7 (50.0%)	0.67	0.13	3.45	0.63
Year of publication	Continuous				1.03	0.9	1.16	0.7
		2009	1 (100.0%)	0 (0.0%)
		2010	1 (33.3%)	2 (66.7%)
		2011	5 (71.4%)	2 (28.6%)
		2012	2 (100.0%)	0 (0.0%)
		2013	6 (66.7%)	3 (33.3%)
		2014	3 (33.3%)	6 (66.7%)
		2015	9 (75.0%)	3 (25.0%)
		2016	5 (55.6%)	4 (44.4%)
		2017	5 (55.6%)	4 (44.4%)
		2018	6 (46.2%)	7 (53.8%)
		2019	3 (75.0%)	1 (25.0%)
		2020	7 (58.3%)	5 (41.7%)
		2021	5 (62.5%)	3 (37.5%)
Number of authors	Continuous	(continuous)			0.93	0.71	1.21	0.59
		2	3 (50.0%)	3 (50.0%)
		3	7 (50.0%)	7 (50.0%)
		4	16 (69.6%)	7 (30.4%)
		5	15 (55.6%)	12 (44.4%)
		6	8 (53.3%)	7 (46.7%)
		7	6 (66.7%)	3 (33.3%)
		8	2 (100.0%)	0 (0.0%)
		9	1 (50.0%)	1 (50.0%)
Conflict of interest reported	Categorical	Yes	32 (56.1%)	25 (43.9%)	0.74	0.32	1.68	0.47
		No	26 (63.4%)	15 (36.6%)	1	-	-	-
Conflict of interest present		Not reported	26 (63.4%)	15 (36.6%)	NA	NA	NA	NA
		No	32 (56.1%)	25 (43.9%)	NA	NA	NA	NA
Funding reported	Categorical	Yes	23 (56.1%)	18 (43.9%)	0.8	0.35	1.81	0.6
		No	35 (61.4%)	22 (38.6%)	1	-	-	-
Type of orthodontic intervention	Categorical	1	41 (59.4%)	28 (40.6%)	1	-	-	-
		2	1 (100.0%)	0 (0.0%)	NA	NA	NA	NA
		3	16 (57.1%)	12 (42.9%)	1.1	0.45	2.67	0.84

Principal findings of the study

This cross-sectional study showed that the majority, i.e., 76.5% (75/98), of the eligible systematic reviews reported or considered (i.e., discussed, weighted) potential adverse effects of orthodontic interventions in the abstract (Table 3). In 40.8% (40/98) spin on adverse effects was found in the abstract of these reviews. Spin related to misleading reporting was the predominant, i.e., 90.0% (36/40), type of spin. (Table 4). No association was found between the presence of spin in the abstract and any of the predictors (Table 5).

Comparison with other studies

Item 1 of the 2004 CONSORT (Consolidated Standards of Reporting Trials) Harms extension [40] states that ‘if the study collected data on harms and benefits, the title or abstract should so state’. In this second part of our cross-sectional studies 76,5% (75/98) of systematic reviews reported or considered adverse effects of interventions in the abstracts. This was lower than the 84.7% (83/98) of reviews that sought and reported findings on adverse effects in the main text or supplementary files of these reviews as we reported in part 1 [33]. A recent overview of systematic reviews by Junqueira et al. [41] found that harms were reported in 47% (258/552) of the abstracts of RCTs published prior to the CONSORT harms statement and in 54% (643/1201) of the abstracts of the RCTs published after the publication of this statement, indicating only a limited improvement in recent years. Qureshi et al. [14] found that most systematic reviews 81.4% (57/70) on interventions with gabapentin reported a statement on harms in the abstract. Different results in reporting of adverse effects in abstracts in this cross-sectional study compared with those in other studies could be the result of variables such as differences in (1) research design, i.e., systematic reviews versus RCTs, (2) sample size (3) what is reported on harms in the abstract, e.g., specific versus more general statements (4) the field of research etc.; For example the relatively high prevalence of reporting of adverse effects in abstracts of systematic reviews in this study could be the result of having included only reviews of orthodontic interventions. Orthodontists might be more attentive in assessing adverse effects, because assessing adverse events such as undesired outcomes of orthodontic treatment and relapse is part of daily clinical practice.

A wide variety of prevalence proportions on spin has been identified in abstracts of systematic reviews of randomized-and non-randomized studies [23,25,28–30]. These studies identified proportions of spin that varied between 23% (24/105) of spin in abstracts of RCTs in rheumatology [28] to 84% (107/128) of spin in abstracts of non-randomized studies that assessed interventions [23]. According to our scoping searches spin in the field of orthodontics has been assessed only in 2 recent studies [31,32]. Spin was identified in 62.2% (69/111) of abstracts of parallel-group RCTs with clearly stated statistically non-significant primary outcomes [31] and in 48.6% (53/109) of abstracts of orthodontic meta-analyses [32]. In our study none of the predictors assessed was associated with the presence of spin in this study. Similar findings were identified by Guo et al. [31] on the overlapping predictors with our study i.e., ‘the year of publication’ and ‘the number of authors’. Makou et al. [32] also found no association with the presence of spin for the overlapping predictors ‘journal’ and ‘year of publication’, but found a higher risk of spin in studies with a large number of authors (≥ 6). However, direct comparisons of our results on spin and those identified in other studies are often difficult because of differences in variables such as (1) the types and subtypes of spin and definitions of spin (2) the research design (3) the locations in the text where spin was assessed (4) the field of research (5) the types of interventions (6) the journals included and (7) the time point of the publication etc. [42].

Strengths and limitations

This study has the following strengths: (1) the research methods were pilot tested on a series of systematic reviews and RCTs to consistently extract data and to calibrate data extractors (2) the protocol of this study was published a priori (3) according to our scoping searches this is the first study that assessed spin on adverse effects in abstracts of systematic reviews of interventions. These searches also showed that our protocol was the first article [24] that planned to assess spin in the field of orthodontics. Subsequently 2 additional studies [31,32] have assessed other types of spin in the orthodontic literature, indicating a growing interest in this topic. (4) all raw data were either included with this manuscript in additional files or registered in Open Science Framework. Open Science Framework (https://osf.io/ka7mp/).

This study also has limitations such as: (1) the risk of inaccurate findings on reporting adverse effects of interventions in abstracts as a result of the inconsistent assessment and reporting of adverse effects in both primary research and in systematic reviews (2) the assessment of spin is not completely objective [42]. However, our inter-operator agreement was high as indicated by a high Cohen’s κ (0.94) and disagreements were completely resolved through discussions (3) the wide variety of different types and definitions of spin and the assessment of spin in different contexts often limits comparing findings on spin between studies [42] (4) this study assessed a variety of proportions exclusively in Cochrane intervention reviews and in the 5 orthodontic journals with the highest impact factor. Our findings therefore probably underestimate the true magnitude of proportions on poor reporting and spin on adverse effects in the abstracts of the wider body of orthodontic systematic reviews and (5) the true magnitude of some proportions could also be underestimated, because we assessed a recent sample (August 1 2009 until July 31 2021) of reviews. In this context, one should consider that poor reporting has decreased over time as was shown in a study that assessed the evolution of poor reporting in 20920 RCTs included in a sample of Cochrane reviews [43].

Implications and future research

Our results imply that end-users of systematic reviews of orthodontic interventions have to be careful when interpreting the findings on adverse effects in abstracts of both Cochrane reviews and those published in the 5 leading orthodontic journals. This is particularly important, because the title and abstracts are often the only read sections of biomedical papers [6] and spin in abstracts can bias the clinician’s interpretations of the results [26]. Reading the full text of research studies is not a solution, because recent studies showed that the proportions of spin in abstracts are similar to those in the full text of the pertinent RCTs [29] and systematic reviews [44]. Guideline developers, researchers, peer reviewers, and editors have an important role in tackling poor reporting and spin regarding adverse effects in abstracts of systematic reviews of orthodontic interventions. Standards for reporting adverse effects in abstracts of systematic reviews of interventions have to be developed. Much research is ahead.

AJODO

American Journal of Orthodontics and Dentofacial Orthopedics

Angle Orthodontist

EJO

European Journal of Orthodontics

KJO

Korean Journal of Orthodontics

O&CR

Orthodontics and Craniofacial Research

MECIR

Methodological Expectations of Cochrane Intervention Reviews

PRISMA

Preferred reporting items for systematic reviews and meta-analyses

RCT

Randomized controlled trial

STROBE

Strengthening the Reporting of Observational Studies in Epidemiology

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Availability of data and material

Not applicable

Competing interests

Reint Meursinge Reynders is an associate editor for Systematic Reviews. All three authors declare that they have no further competing interests.

Funding

All expenses for developing and conducting of this cross-sectional study were paid evenly by each author.

Authors' contributions

This cross-sectional study was conceived and designed by PS and RMR. These 2 researchers also conducted all data extraction procedures. PS, RMR, and NDG conducted the data analyses and statistical analyses. NDG provided support on methodological issues. All 3 authors revised the various draft versions of this paper and read and approved the final submitted manuscript. RMR is the guarantor.

Acknowledgements

Not applicable

Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, Gøtzsche PC, Lasserson T, Tovey D; PRISMA for Abstracts Group. PRISMA for Abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10(4):e1001419. doi: 10.1371/journal.pmed.1001419. Epub 2013 Apr 9. PMID: 23585737; PMCID: PMC3621753.
Haneef R, Lazarus C, Ravaud P, Yavchitz A, Boutron I. Interpretation of Results of Studies Evaluating an Intervention Highlighted in Google Health News: A Cross-Sectional Study of News. PLoS One. 2015 Oct 16;10(10):e0140889. doi: 10.1371/journal.pone.0140889. eCollection 2015.
Haneef R, Yavchitz A, Ravaud P, Baron G, Oranksy I, Schwitzer G, Boutron I. Interpretation of health news items reported with or without spin: protocol for a prospective meta-analysis of 16 randomised controlled trials. BMJ Open. 2017 Nov 17;7(11):e017425. doi: 10.1136/bmjopen-2017-017425.
Ad Hoc Working Group for Critical Appraisal of the Medical Literature (1987). A proposal for more informative abstracts of clinical articles. Ann Intern Med. 106: 598–604.
Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ (1990) More informative abstracts revisited. Ann Intern Med. 113: 69–76.
Dogan RI, Murray GC, Névéol A, Lu Z. Understanding PubMed user search behavior through log analysis. Database (Oxford). 2009;2009:bap018. Epub 2009 Nov 27.
Page MJ, Cumpston M, Chandler J, Lasserson T. Chapter III: Reporting the review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventionsversion 6.2 (updated February 2021). Cochrane, 2021. Available from training.cochrane.org/handbook.
Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R. Methodological Expectations of Cochrane Intervention Reviews. Cochrane: London, Version February 2021. [online] Available from: MECIR Version February 2021 clean final_1.pdf (cochrane.org) (accessed December 4th 2021).
Bagul NB, Kirkham JJ. The reportingof harms in randomized controlled trials of hypertension using the CONSORT criteria for harm reporting. Clin Exp Hypertens. 2012;34(8):548-54. doi: 10.3109/10641963.2012.681724. Epub 2012 May 9.
Carmichael K, Nolan SJ, Weston J, Tudur Smith C, Marson AG. Assessment of the quality of harms reporting in non-randomised studies and randomised controlled studies of topiramate for the treatment of epilepsy using CONSORT criteria. Epilepsy Res. 2015 Aug;114:106-13. doi: 10.1016/j.eplepsyres.2015.04.019. Epub 2015 May 8. PMID: 26088893.
Hodkinson A, Kirkham JJ, Tudur-Smith C, Gamble C. Reportingof harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension. BMJ Open. 2013 Sep 27;3(9):e003436. doi: 10.1136/bmjopen-2013-003436.
Parsons R, Golder S, Watt I. More than one-third of systematic reviews did not fully report the adverse events outcome. J Clin Epidemiol.2019 April;108:95-101. org/10.1016/j.jclinepi.2018.12.007.
Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med.2009 Oct 26;169(19):1756-61.
Qureshi R, Mayo-Wilson E, Rittiphairoj T, McAdams-DeMarco M, Guallar E, Li T. Summaries of harms in systematic reviews are unreliable Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin, Journal of Clinical Epidemiology (2021), doi: https://doi.org/10.1016/j.jclinepi.2021.10.024
Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014 Nov 21;349:g6501. doi: 10.1136/bmj.g6501. PMID: 25416499; PMCID: PMC4240443.
Zorzela L, Golder S, Liu Y, Pilkington K, Hartling L, Joffe A, Loke Y, Vohra S. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014 Jan 8;348:f7668. doi: 10.1136/bmj.f7668. PMID: 24401468; PMCID: PMC3898583.
Cochrane library. Cochrane database of systematic reviews. [online] Available from: About Cochrane Reviews | Cochrane Library (accessed December 8th 2021).
Steegmans PAJ, Bipat S, Meursinge Reynders RA. Seeking adverse effects in systematic reviews of orthodontic interventions: protocol for a cross-sectional study. Syst Rev. 2019 Apr 5;8(1):89. doi: 10.1186/s13643-019-1000-1. PMID: 30953538; PMCID: PMC6449933.
Glossary of terms in the Cochrane Collaboration. Version 2.5. Updated May 2005. [online] Available from: http://aaz.hr/resources/pages/57/7.%20Cochrane%20glossary.pdf (accessed December 4th 2021).
Peryer G, Golder S, Junqueira D, Vohra S, Loke YK. Chapter 19: Adverse effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from training.cochrane.org/handbook.
Boutron I, Ravaud P. Misrepresentation and distortion of research in biomedical literature. Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2613-2619. doi: 10.1073/pnas.1710755115.
Boutron I. Spin in Scientific Publications: A Frequent Detrimental Research Practice. Ann Emerg Med. 2020 Mar;75(3):432-434. doi: 10.1016/j.annemergmed.2019.11.002. Epub 2019 Dec 23. PMID: 31874770.
Lazarus C, Haneef R, Ravaud P, Boutron I. Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention. BMC Med Res Methodol. 2015 Oct 13;15:85. doi: 10.1186/s12874-015-0079-x.
Steegmans, P.A.J., Di Girolamo, N, Meursinge Reynders, R.A. Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews. Res Integr Peer Rev4, 27 (2019). https://doi.org/10.1186/s41073-019-0084-4.
Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. 2010 May 26;303(20):2058-64. doi: 10.1001/jama.2010.651.
Boutron I, Altman DG, Hopewell S, Vera-Badillo F, Tannock I, Ravaud P. Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol. 2014 Dec 20;32(36):4120-6. doi: 10.1200/JCO.2014.56.7503. Epub 2014 Nov 17. PMID: 25403215.
Yavchitz A, Ravaud P, Altman DG, Moher D, Hrobjartsson A, Lasserson T, Boutron I. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol. 2016 Jul;75:56-65. doi: 10.1016/j.jclinepi.2016.01.020. Epub 2016 Feb 2.
Mathieu S, Giraudeau B, Soubrier M, Ravaud P. Misleading abstract conclusions in randomized controlled trials in rheumatology: comparison of the abstract conclusions and the results section. Joint Bone Spine. 2012 May;79(3):262-7. doi: 10.1016/j.jbspin.2011.05.008. Epub 2011 Jul 5. PMID: 21733728.
Khan MS, Lateef N, Siddiqi TJ, Rehman KA, Alnaimat S, Khan SU, Riaz H, Murad MH, Mandrola J, Doukky R, Krasuski RA. Level and Prevalence of Spin in Published Cardiovascular Randomized Clinical Trial Reports With Statistically Nonsignificant Primary Outcomes: A Systematic Review. JAMA Netw Open. 2019 May 3;2(5):e192622. doi: 10.1001/jamanetworkopen.2019.2622. PMID: 31050775; PMCID: PMC6503494.
Jones C, Rulon Z, Arthur W, Ottwell R, Checketts J, Detweiler B, Calder M, Adil A, Hartwell M, Wright DN, Vassar M. Evaluation of spin in the abstracts of systematic reviews and meta-analyses related to the treatment of proximal humerus fractures. J Shoulder Elbow Surg. 2021 Jan 19:S1058-2746(21)00016-1. doi: 10.1016/j.jse.2020.11.026. Epub ahead of print. PMID: 33482369.
Guo F, Fang X, Li C, Qin D, Hua F, He H. The presence and characteristics of 'spin' among randomized controlled trial abstracts in orthodontics. Eur J Orthod. 2021 Oct 4;43(5):576-582. doi: 10.1093/ejo/cjab044. PMID: 34397084.
Makou O, Eliades T, Koletsi D. Reporting, interpretation, and extrapolation issues (SPIN) in abstracts of orthodontic meta-analyses published from 2000 to 2020. Eur J Orthod. 2021 Mar 19:567-575. doi: 10.1093/ejo/cjab009. Epub ahead of print. PMID: 33740054.
Steegmans 2022 submitted to Systematic Reviews: Seeking adverse effects in systematic reviews of orthodontic interventions: a cross-sectional study (Part 1).
Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, Ravaud P. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308. doi: 10.1371/journal.pmed.1001308. Epub 2012 Sep 11. PMID: 22984354; PMCID: PMC3439420.
von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007 Oct 16;147(8):573-7.
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from training.cochrane.org/handbook.
The Cochrane Database of Systematic Reviews(CDSR). [online] Available from: https://www.cochranelibrary.com/cdsr/about-cdsr (accessed December 4th 2021).
Preoteasa CT, Ionescu E, Preoteasa E. Chapter 18: Risks and complications associated with orthodontic treatment. In: Bourzgui F. (editor). Orthodontics-Basic aspects and clinical considerations. March 9, 2012 under CC BY 3.0 license. www.intechopen.com. [online] Available from: https://cdn.intechopen.com/pdfs/31388/InTech-Risks and complications associated with orthodontic treatment.pdf (accessed December 4^th 2021).
Sergeant ESG. Sergeant, ESG, 2018. Epitools Epidemiological Calculators. Ausvet. [online] Available from: http://epitools.ausvet.com.au. (accessed December 4^th 2021).
Ioannidis JP, Evans SJ, Gøtzsche PC, O'Neill RT, Altman DG, Schulz K, Moher D; CONSORT Group. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med.2004 Nov 16;141(10):781-8.
Junqueira DR, Phillips R, Zorzela L, Golder S, Loke Y, Moher D, Ioannidis JPA, Vohra S. Time to improve the reporting of harms in randomized controlled trials. J Clin Epidemiol. 2021 Aug;136:216-220. doi: 10.1016/j.jclinepi.2021.04.020. Epub 2021 May 10. PMID: 33984494.
Chiu K, Grundy Q, Bero L. 'Spin' in published biomedical literature: A methodological systematic review. PLoS Biol. 2017 Sep 11;15(9):e2002173. doi: 10.1371/journal.pbio.2002173. PMID: 28892482; PMCID: PMC5593172.
Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, Perrodeau E, Altman DG, Ravaud P. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ. 2017 Jun 8;357:j2490. doi: 10.1136/bmj.j2490. Erratum in: BMJ. 2017 Aug 8;358:j3806. PMID: 28596181.
Eleftheriadi I, Ioannou T, Pandis N. Extent and prevalence of spin in randomized controlled trials in dentistry. J Dent. 2020 Sep;100:103433. doi: 10.1016/j.jdent.2020.103433. Epub 2020 Jul 17. PMID: 32687962.

Additionalfile1STROBEchecklist.docx
Additional file 1.Format: docxTitle and description: Additional file 1. STROBE Checklist: Checklist of items that should be included in reports of cross-sectional studies [35]
Additionalfile2.docx
Additional file 2. Format: docxTitle and description: Additional file 2.
Additionalfile3.docx
Additional file 3.Format: docxTitle and description: Additional file 3. Included reviews

Download PDF

Journal Publication

published 20 Jun, 2023

Read the published version in Systematic Reviews →

Editorial decision: Major revision
26 Jan, 2023
Reviewers agreed at journal
12 Jun, 2022
Reviewers invited by journal
09 May, 2022
Editor assigned by journal
04 May, 2022
First submitted to journal
11 Feb, 2022

You are reading this latest preprint version

Spin on adverse effects in abstracts of systematic reviews of orthodontic interventions: A cross-sectional study (Part 2)

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Objectives

Methods

Results

Discussion

Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1