Research transparency promotion by surgical journals publishing randomised controlled trials: a survey

doi:10.21203/rs.2.15686/v3

Download PDF

Research

Research transparency promotion by surgical journals publishing randomised controlled trials: a survey

https://doi.org/10.21203/rs.2.15686/v3

This work is licensed under a CC BY 4.0 License

Journal Publication

published 01 Oct, 2020

Read the published version in Trials →

You are reading this older preprint version

Read the latest preprint version →

Objective: To describe the surgical journal position statement on data-sharing policies (primary objective) and to describe the other features of their research transparency promotion.

Methods: Only “SURGICAL” journals with an impact factor superior to 2 (Web of Science) were eligible for the study. They were not included if there were no explicit instructions for clinical trial publication in the official instructions for authors (OIA) and if there were no randomized controlled trial (RCT) published between 1st January 2016 and 31st December 2018. The primary outcome was the existence of a data-sharing policy in the instructions for authors. Data sharing policy was detailed in 3 categories, inclusion of data sharing policy mandatory, optional or not available. Details on research transparency promotion were also collected, namely the existence of a “prospective registration of clinical trials requirement” policy; a conflict of interests (COIs) disclosure requirement and a specific reference to reporting guidelines such as CONSORT for RCT.

Results: Among the 87 surgical journals eligible, 82 were included in the study: 67 (82%) had explicit instructions for RCT and of the remaining, 15 (18%) had published at least one RCT. The median impact factor was 2.98 [IQR=2.48-3.77] and in 2016 and 2017, the journals published a median of 11.5 RCT [IQR=5-20.75]. Data-sharing statement instructions (primary outcome) was retrieved in 41 journals (50%); Data-sharing statement instructions were mandatory in four cases (4.9%), optional in 45% (n=37) and not available in 50% (n=41) of the journals. As for data-sharing statements, no association was found between journal characteristics and the existence of data-sharing policies (mandatory or optional). A “prospective registration of clinical trials requirement” was associated with ICMJE allusion or affiliation and higher impact factors. Journals with specific RCT instructions in their OIA and journals referenced on the ICMJE website more frequently mandated the use of CONSORT guidelines.

Conclusion: Research transparency promotion is still limited in surgical journals. Standardisation of journal requirements according to ICMJE guidelines could be a first step forward for research transparency promotion in surgery.

General Surgery

Translational Medicine

Data sharing

Conflict of interest

CONSORT statement

Randomised controlled trial

Surgical journals have a key role to ensure transparency, openness, and reproducibility [1] to increase value and reduce waste in the research they publish [2]. Editorial standards promoting transparency are expected when it comes to randomised controlled trials (RCT) because their importance is paramount in drafting guidelines that can impact medical practice worldwide. Surgical interventions are invasive and in some surgical trials, participants may expose themselves to heightened risk with uncertain benefits. It results an implicit social contract imposing an ethical obligation that the results lead to the greatest possible benefit to society [3].

The latest breakthrough was the adoption by the International Committee of Medical Journal Editors (ICMJE) of a policy that encourages RCT data sharing and requires a data-sharing statement to be included in the reports of published clinical trials [4]. According to the ICJME requirement, the data sharing statement must indicate if individual anonymize data will be shared. The statement must specify from when and until when will the data be available (start and end date). Authors have to clarify with whom data may be shared, the modality of access and what other documents would be available.

Other aspects of research transparency promotion have been previously promoted such as registration of the trial [5], adoption of the CONSORT statement [6] and declaration of conflict of interest (COI) [7]. However, transparent practices in the surgical community could be suboptimal as suggested by the underreporting of COI [8].

If we hypothesize that editors can be the first motivators for research’ transparency promotion, reviewing current editors‘ practices and policies in relation to the transparency of the research is pertinent before evaluate the evolution of those practices in papers published in surgical journals.

The aim of this study is to describe the surgical journal position statement on data-sharing policies (primary objective) and to describe the other features of their research transparency promotion.

This survey of surgical journals was registered with a protocol in the Open Science Framework (OSF) on February, 25^th 2019 [9].

Eligibility criteria and journal selection

Two reviewers (NL and AG) used Web of Science to select journals classified in the "SURGICAL" category with a 2017 impact factor superior to 2. Surgical journals were not included if there were no explicit instructions for clinical trial publication in the instructions for authors and if there were no RCT published between 1^st January 2016 and 31^st December 2018. Two authors (NL and AG) independently extracted the data. Disagreements were resolved by consensus or in consultation with a third reviewer (DB). The list of journals was extracted in December 2018 and the official instructions for authors (OIA) were downloaded on January 13-14, 2019.

Our primary outcome was the existence of a data-sharing policy in the instructions for authors. Types of policies for data sharing were described using the following classification: “Inclusion of data sharing policy mandatory”, “inclusion of data sharing policy optional” and “no data sharing policy available”.

Other details on research transparency promotion were also collected, namely the existence of a “prospective registration of clinical trials requirement” policy; a “COIs” disclosure requirement and a specific reference to reporting guidelines such as CONSORT for RCT.

Other variables were also extracted, first to detail journals features and also as potential predictors for journals adherence to research transparency rules. To be done, 2017 journal impact factors were extracted from the Web of Science data base and the number of RCT published in 2016 and 2017 (1/1/16 to 12/31/17) was extracted from Pubmed. As exposed in the protocol, it was originally planned to extract the number of RCT published in 2016, 2017 and 2018 but this was not possible because at the time of data extraction, all RCT published were not fully indexed in Pubmed because of various deadlines indexing between different journals and publishers. ICMJE “affiliation” was defined as journals referenced as "Journals stating that they follow ICMJE Recommendations" at: http://www.icmje.org/journals-following-the-icmje-recommendations. Publishing model was classified according to the details exposed in the instructions to authors about who has to paid for publication: the authors (open access), reader (subscription to the journal or pay per view) and optional (can be the author, equivalent to open access or the reader).

Statistical analyses

Analyses of all included journals were performed using R statistical software (http://www.r-project.org/). Continuous data were expressed with median and the interquartile range (IQR) and compared with the Mann-Whitney U test. Categorical data were expressed as a percentage and compared with a Chi-squared test or a two-sided Fisher test when Chi-squared test application conditions were not met. Univariate exploratory analyses were performed to explore the associations between journal features and the various transparency policies. For exploratory analyses, some data arrangements have been made. Publishers were grouped into 3 classes according to the number of journal titles included in this study: “high” with over 20 titles, “middle” with 10 to 20 titles and “low” with less than 10 titles. Number of RCT published and 2017 journal impact factor were separated into quartile using the quant.cut () function including the lowest variable and excluded the right. For exploratory analyses, the threshold for statistical significance was set at P < 0.05/36 (P=0.0014) with a Bonferroni correction because of multiple comparison (n=36). Multivariable analysis was not performed due to sparse data.

Among the 87 surgical journals eligible, 82 were included in the analysis: 67 (82%) had explicit instructions for RCT and of the remaining, 15 (18%) had published at least one RCT between 2016-2018 (Figure 1). The characteristics of these journals are detailed in Table 1. The median impact factor was 2.98 [IQR=2.48-3.77]. In 2016 and 2017, the journals published a median of 12 RCT [IQR=5-21]. The publishing model was “optional” in most cases (89%) and America was the principal geographical area of journal editorial committees (56%).

Adherence to research transparency rules are details in the Table 2. Data-sharing statement instructions were retrieved in 41 OIA’s journal (50%). In OIA’s journals, inclusion of a data sharing statement was mandatory in four cases (5 %), optional in 45% (n=37) and not available in 50% (n=41) of the journals. COI disclosure was mandatory in 77 journals (94%). A reference to CONSORT guidelines was made in 24 journals (29%). Prospective registration of clinical trials was mandatory in 53 cases (65%). Table 3 detailed the relationship between data sharing policies and journal impact factors and the number of RCT published in 2016 and 2017 grouped by quartiles. Other research transparency promotion items relationship with journal impact factor and the number of RCT published are illustrated in the Figure 2. Except the association between journal impact factor and trial registration mandatory rule (P=0.003), no relevant association was found. Other exploratory analysis concerning journal features and the different transparency policies are presented in Table 4. As for data-sharing statements, no association was found between journal characteristics and the existence of data-sharing policies (mandatory or optional). A “prospective registration of clinical trials requirement” was associated with ICMJE allusion (P<0.001), ICMJE affiliation (P<0.001) and higher impact factors (P<0.001). Journals with specific RCT instructions in their OIA (P=0.04) and journals referenced on the ICMJE website (P=0.03) more frequently mandated the use of CONSORT guidelines but those results were not significant with our P value threshold (P=0.0014). No other pertinent association was found.

We noted low rates of implementation of data-sharing policies, i.e. 50 % of the journals had no explicit policy included in their instructions for authors. When explicit, these policies were rather optional contrary to the ICMJE recommendation that make a data-sharing statement mandatory for RCT. Of course, the study was performed during a time of change and one could argue that the ICMJE position on data sharing was fairly recent (data extracted 6 months after the ICMJE statement) and that a number of journals did not have the time to implement it when our survey was conducted. However, this policy was announced in 2017 [4] and 35 (43%) journals are listed on the ICMJE website. Interestingly, implementation of older policies was also suboptimal, even for making a specific reference to reporting guidelines such as CONSORT for RCT which date from 1996 [6]. Except for COI disclosure, those policies were mostly non-binding. These disappointing results are not new. In 2014, Chapman et al. [10] warned about sub-optimal transparency policies in 10 leading surgical journals.

We considered a journal’s policies presented on its website as a surrogate marker of implementation of these policies. However, it is possible that editors of journals with a policy do not implement them in an optimal manner [11] or, conversely, that a journal with no specific policy documented on the website requires authors to comply with some of the features we explored. Of note, previous research has shown that journal requirements can have a significant impact on changing researcher practices [12], an obvious next step will be to explore the transparency features of the published RCT in these journals.

Of concern, we found no association of research transparency items with impact factors nor with the number of RCT published except for prospective trial registration among the surveyed surgical journals. This is of concern since impact factor is misused as a surrogate to assess the quality of a given journal and sometimes of an individual paper. [13,14]

Some limitations of our study have to be underlined. First, we want to insist that the goal of the study was to evaluate transparency research promotion by surgical journals through their official instruction for authors. Consequently, no conclusion about a relation between our results and the quality of published papers in terms of transparency can be drawn because of the design of our study. Future research about journals or publishers’ policies impact on published papers will be important to assess the potential impact of such policies on reporting. Second, we have to underline that the official registration of our study protocol on OSF after the initiation of the study even if it was written before and not modified since (author’s statement). The completely descriptive aspect of our main outcome limits the potential bias in this situation. At last, we must point out that we have chosen arbitrary to combined data sharing policies mandatory and optional based on the verbatim analyses we made during extraction. Data sharing statement requirement were rarely mandatory but, in most cases, only optional. For exploratory analyses (journal features associated to research transparency promotion items), we had the feeling that journals suggesting data sharing wasn't so bad considering how far behind other, much older policies are which is why we combined the 2 variables (mandatory and optional).

Data sharing policies appear to be sub optimally adopted and promoted by surgical journals. We suggest that indicators of quality such as prospective audits of policies and published papers must be used to assess journals instead of journal impact factors. We encourage surgical journals to be part of the move to improve their research transparency promotion. Standardisation of journal requirements according to ICMJE guidelines could be a first step forward for research transparency promotion in surgery.

Ethics approval and consent to participate: For this type of study, formal consent is not required.

Consent for publication: Not applicable

Availability of data and material: Study protocol is already available on Open Science Framework. Data extracted and statistical code is available on Open Science Framework.

Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) No authors have support from any company for the submitted work; (2) None has relationships (travel/accommodations expenses covered/reimbursed) who might have an interest in the work submitted in the previous three years. None have no relationship with any company that might have an interest in the work submitted; (3) no author’s spouse, partner, or children have any financial relationships that could be relevant to the submitted work; and (4) none of the authors has any non-financial interests that could be relevant to the submitted work.

Funding: There was no financial support for this study. Florian Naudet's works on reproducibility are part of the ReiTheR (reproducibility in therapeutic research) project. This project is funded by the French National Research Agency (ANR, reference number ANR-17-CE-36-0010-01). The sponsor had no role concerning preparation, review, or approval of the manuscript.

Author’s contributions

Study concept and design: DB, FN

Acquisition of data: NL, AG, DB

Analysis and interpretation of data: DB, NL, FN

Drafting of the manuscript: DB, NL, FN

Critical revision of the manuscript for important intellectual content: FN, KB, LS, AG

Statistical analysis: DB, FN

Acknowledgements: None

Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, et al. Promoting an open research culture. Science. 2015;348:1422–5.
Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, et al. Biomedical research: increasing value, reducing waste. The Lancet. 2014;383:101–4.
Bauchner H, Golub RM, Fontanarosa PB. Data Sharing: An Ethical and Scientific Imperative. JAMA. 2016;315:1238.
Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data Sharing Statements for Clinical Trials - A Requirement of the International Committee of Medical Journal Editors. N Engl J Med. 2017;376:2277–9.
De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250–1.
Limb C, White A, Fielding A, Lunt A, Borrelli MR, Alsafi Z, et al. Compliance of Randomized Controlled Trials Published in General Surgical Journals With the CONSORT 2010 Statement. Ann Surg. 2019;269:e25–7.
Conflict of interest. International Committee of Medical Journal Editors. Ann Intern Med. 1993;118:646–7.
Ziai K, Pigazzi A, Smith BR, Nouri-Nikbakht R, Nepomuceno H, Carmichael JC, et al. Association of Compensation From the Surgical and Medical Device Industry to Physicians and Self-declared Conflict of Interest. JAMA Surg. 2018;153(11):997-1002
Bergeat D, Nicolas L, Anis G, Florian N. Research Transparency Promotion by Surgical Journals [Internet]. OSF; 2019. Available from: osf.io/gkc7f
Chapman SJ, Shelton B, Mahmood H, Fitzgerald JEF, Harrison E, Bhangu A. Promoting transparency in clinical research: Systematic review of disclosure and data-sharing policies in surgical journals. Int J Surg. 2014;12:S52.
Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ. 2018;360:k400.
Giofrè D, Cumming G, Fresc L, Boedker I, Tressoldi P. The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PloS One. 2017;12:e0175583.
Ioannidis JPA, Thombs BD. A user’s guide to inflated and manipulated impact factors. Eur J Clin Invest. 2019;49:e13151.
San Francisco Declaration on Research Assessment. Available from: https://sfdora.org/read/

RCT: Randomised Controlled Trial

ICMJE: International Committee of Medical Journal Editors

COI: Conflict of interest

OIA: Official Instructions for Authors

IQR: Interquartile interval range

Due to technical limitations, tables are only available as a download in the supplemental files section

SuppTable1.xlsx
SuppFig1final.jpg
Table1JDSprojectR2allmarked.xlsx
Journal characteristics and policies. OIA = official instructions for authors, ICMJE = International Committee of Medical Journal Editors.
Table2R2allmarked.xlsx
Adherence to research transparency rules. COI = Conflict of interests, ICMJE = International Committee of Medical Journal Editors.
Table3JDSR2.xlsx
Relationship between data sharing policies and journal impact factors and the number of RCT during the study period grouped by quartiles.
Table4R2allmarked.xlsx
Exploratory analysis, association between journal features and transparency policies with univariate analysis. OIA = official instructions for authors, ICMJE = International Committee of Medical Journal Editors. Significant P value (P<0.0014) are highlighted in bold and underlined.

Download PDF

Journal Publication

published 01 Oct, 2020

Read the published version in Trials →

Editorial decision: Minor revision
24 Aug, 2020
Review #2 received at journal
23 Aug, 2020
Reviewer #2 agreed at journal
18 Aug, 2020
Review #1 received at journal
18 Aug, 2020
Reviewer #1 agreed at journal
11 Aug, 2020
Reviewers invited by journal
10 Aug, 2020
Editor assigned by journal
07 Aug, 2020
Submission checks completed at journal
06 Aug, 2020

You are reading this older preprint version

Read the latest preprint version →

Research transparency promotion by surgical journals publishing randomised controlled trials: a survey

Status:

Journal Publication

Version 3

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Declarations

References

List Of Abbreviations

Tables

Supplementary Files

Status:

Journal Publication

Version 3