Rigorous Reporting of Stepped Wedge Cluster Randomised Controlled Trials: Development of a Flow Diagram Template

doi:10.21203/rs.2.39/v1

Download PDF

Method Article

Rigorous Reporting of Stepped Wedge Cluster Randomised Controlled Trials: Development of a Flow Diagram Template

https://doi.org/10.21203/rs.2.39/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Clear, unambiguous reporting of a stepped-wedge cluster randomised controlled trials (SW-CRCTs) is a challenge, requiring attention to at least five reporting guidelines. A recent systematic review on the reporting quality of SW-CRCTs noted that only 24 out of their 39 included studies (61.5%) provided a flow diagram. The available flow diagram templates may be unable to accommodate the range of SW-CRCT designs and reporting requirements, thus hampering reporting standards. We aimed to review the degree to which current flow diagrams used in SW-CRCT reports support comprehensive reporting to standards. In the absence of a suitable model we aimed to develop a flexible SW-CRCT flow diagram template that would support reporting standards and facilitate transparency in the description of our SW-CRCT.

Methods

We identified flow diagrams used in the reporting of 24 SW-CRCTs. We extracted data relating to 22 pre-determined reporting criteria derived from six reporting guidelines relevant to a SW-CRCT that we were conducting. Data were extracted independently by two reviewers entered into Microsoft® Excel and analysed using descriptive statistics and narrative summary.

Results

None of the 24 flow diagrams met all 22 criteria. Nine (38%) reported data only at participant level, three (13%) only at cluster level, 10 (42%) attempted to report on both participant and cluster levels on a single flow diagram, and two of the trials reported information over two separate flow diagrams. Reporting was better at participant rather than cluster level, which was very limited.

Conclusions

Our review of published SW-CRCT flow diagrams has highlighted several limitations. Inflexible models fail to support the reporting of all expected information items to agreed standards. We acknowledge that it is unlikely that one flow diagram model will be applicable to all SW-CRCTs; however, the flexible flow diagram template we developed was used to good effect in our study and ensured that relevant reporting items were elucidated.

Trial registration

ClinicalTrial.gov NCT01954212. Registered 21st August 2013

General Medicine

In recent years numerous reporting guidelines (and extensions) have been published for use when writing, reviewing, or assessing reports for the purposes of improving the quality, transparency and comprehensiveness of trial reports (1)(Moher et al, 2001). The original Consolidated Standards of Reporting Trials (CONSORT) Statement (2)(Begg et al 1996; revised by Moher et al 2001), includes a checklist of items that should be included in any trial report and a diagram to show the flow of participants from group assignment through to intervention and treatment, outcome data, losses to the trial and the final analysis.

However, it has been recognised that the reporting of cluster randomised controlled trials (CRCTs) is more complex and requires additional information to that of parallel group randomised trials to allow readers to interpret them accurately (3)(Campbell et al, 2004). The original CONSORT flow diagram is not sufficient for CRCTs, so an extended CONSORT Statement was published by Campbell et al in 2004 and revised in 2012 (4). More recently, motivated by the unique design characteristics of SW-CRCTs, a CONSORT extension for the SW-CRCT was developed and published (5).

A template flow diagram is provided in the CONSORT CRT extension, acknowledging that its exact form and content will vary in relation to the specific features of a trial and the type of analysis (4)(Campbell, 2012), and a specimen flow diagram is presented in the SW-CRCT extension (5). If the analysis is multilevel or hierarchical, the flow diagram should present the flow of both clusters and individuals through the trial, from assignment to analysis. Specifically, the flow diagram should elucidate the numbers of clusters that were randomly assigned, received intended treatment, and were analysed for the primary outcome, and include the losses and exclusions for both clusters and individual cluster members (4)(Campbell, 2012). Hemming et al (6)(2015) recommend that a flow diagram should illustrate how many participants are within each cell of the study design.

Although it appears that use of CONSORT checklists and flow diagrams has been associated with some improvement in reporting of parallel group randomised trials (7)(Plint et al, 2006), limitations in the quality of the reporting of SW-CRCTs remain (8)(Beard et al 2015). Martin et al (9)(2016) in the first systematic review of SW-CRCTs to assess reporting adherence to CONSORT items highlighted that the quality of reporting in stepped-wedge trials is suboptimal, for example, in relation to type of design and sample size calculations.

A more recent systematic review by Grayling et al (10)(2017) that assessed the reporting quality of SW-CRCTs across all sections of the CONSORT statement found that, although the standard of reporting has improved since 2010, the standard of reporting varied significantly in quality. The percentage of trials reporting each criterion varied to as low as 15.4%, with a median of 66.7%. Furthermore, they noted that only 24 out of their 39 included studies (61.5%) provided a flow diagram.

We recently conducted a Phase II pilot SW-CRCT (SOCLE II), which explored the impact of a complex oral healthcare (OHC) intervention after stroke (11)(Brady et al, 2015). The clusters within this study were four stroke care wards (sites), and the intervention was delivered within each cluster with outcome data collected from both patient and staff participants. Site level cluster randomisation progressed in a stepped wedge manner, where, after a period of baseline data collection, each site converted from usual care to the enhanced OHC intervention in a randomly allocated order.

We wrestled with the challenge of ensuring our trial reported to all the various recommended criteria from current trial reporting standards within one flow diagram. Diagram templates within the reporting standards did not support reporting all of the required details. Furthermore, the limited adherence to the reporting standards identified in the aforementioned systematic review indicated that current flow diagram models may provide inadequate support in reporting to the various reporting standards.

We aimed to review the quality of flow diagrams used in previous SW-CRCTs and, where a suitable template was unavailable, we aimed to develop a flexible SW-CRCT flow diagram that could accommodate the relevant reporting standards and facilitate transparency in SW-CRCT reports.

As a basis for our SW-CRCT flow diagram evaluation we used the included trial reports in the Grayling et al (10)(2017) systematic review. This comprehensive review of reports of original ‘classical’ SW-CRCTs included searches of PubMed (including Medline), Ovid (including Embase), Web of Knowledge, PsycINFO, the Cochrane Library, the ISRCTN registry and ClinicalTrials.gov up until 24 February 2015, with no publication date restrictions. Having initially found 7720 records they identified 39 SW-CRCT reports, of which only 24 included a flow diagram(s).

For the purposes of our review, studies were eligible for inclusion if they were reports on a completed SW-CRCT and that included one or more flow diagrams. All 24 full text articles cited in the Grayling et al (10)(2017) review that included a flow diagram(s) were retrieved and stored. Study authors were not required to be contacted as we were only concerned with the information that was explicitly reported in the published flow diagrams.

Each of the 24 papers’ flow diagrams were reviewed against 22 pre-determined criteria that we identified from across six key reporting standards that were relevant to our SOCLE II SW-CRCT. These included the: Consolidated Standards for Reporting Trials (CONSORT) (12)(Schulz et al 2010); relevant extensions for cluster and pragmatic randomised controlled trials (4,13)(Campbell et al 2012; Zwarenstein et al 2008); pilot randomised controlled trials (14)(Eldridge et al 2016); Guidelines for STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) (15)(von Elm et al, 2014); and the extension specific to a RCT of a non-pharmacological intervention (16)(Boutron et al 2008) (Table 1).

Table 1 Reporting Standards relevant to the SOCLE II Study

Reporting Standard	Relevance to the SOCLE II Study	Sample of key reporting Requirements in relation to the flow diagram
CONSORT 2010 Statement	Standard for all RCT reports	13a For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome 13b For each group, losses and exclusions after randomisation, together with reasons
CONSORT Cluster Extension 2012	SOCLE II was a SW-CRCT	15a For each group, the numbers of clusters that were randomly assigned, received intended treatment, and were analysed for the primary outcome 15b For each group, losses and exclusions for both clusters and individual cluster members
CONSORT Pragmatic Trials Extension 2008	SOCLE II was a pragmatic trial	The number of participants or units approached to take part in the trial, the number which were eligible, and reasons for non-participation should be reported
CONSORT Extension to Pilot RCTs & Feasibility Trials 2016	SOCLE II was a pilot RCT	For each group, the numbers of participants who were approached and/or assessed for eligibility, randomly assigned, received intended treatment, and were assessed for each objective
STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Observational studies Guidelines 2007	Stepped Wedge trials are closely related to before/ after studies	Report numbers of individuals at each stage of study, e.g. numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed Give reasons for non-participation at each stage
CONSORT Extensions for Non-pharmacological RCTs 2008	SOCLE II was a non-pharmacological intervention	The number of care providers or centres performing the intervention in each group and the number of patients treated by each care provider or in each centre

The 22 pre-determined quality criteria the numbers against which each flow diagram was evaluated, 11 each for participants and clusters, were: (i) approached and (ii) eligible to take part; (iii) reported reasons for non-participation at each stage; (iv) randomly assigned; (v) in receipt of the intended treatment; (vi) analysed for each objective; (vii) analysed for the primary outcome (viii); lost and (ix) excluded; (x) included in the study and (xi) completed follow-up.

One researcher (BB) extracted the data represented in each flow diagram against these pre-determined criteria entering all data into Microsoft® Excel. A second reviewer (HG) conducted a validation of the extraction process and uncertainties were resolved through discussion. Data were analysed using descriptive statistics and narrative summary.

We reviewed 24 full text trial reports that were all in English. We found that none of the 24 flow diagrams reported to the 22 reporting criteria standards. Nine (38%) of the diagrams reported only at participant level; three (13%) only at cluster level; 10 (42%) attempted to report on both participant and cluster levels on a single flow diagram; and the remaining two trials reported information over two separate flow diagrams (see supplementary data file for details).

Overall, only six (25%) of the diagrams met at least half of the 22 pre-determined criteria, with the most comprehensive flow diagram meeting 13/22 (59%) and one diagram only meeting 2/11 (9%) of the criteria.

Reporting was better at participant rather than cluster level, with 19/24 (79%) reporting the number of participants eligible to take part; 18/24 (75%) reporting the number who received intended treatment; 15/24 (63%) reporting losses and exclusions; 14/24 (58%) reporting the numbers approached; 13/24 (54%) reporting reasons for non-participation; and 13/24 (54%) reporting numbers analysed for primary outcome. However, only 6/24 (25%) reported on the random assignment of participants and ‘treated by each care provider or in each care centre’, and only one study reported that participants were ‘assessed for each objective’.

Information at cluster level was very limited; the most frequently reported criteria were the numbers of clusters that were randomly assigned 13/24 (54%), performed the intervention 12/24 (50%), received intended treatment 10/24 (42%), eligible to take part 8/24 (33%), and approached to take part 7/24 (29%). There was minimal reporting ‘assessed for each objective’ 1/24 (4%); reasons for non-participation 5/24 (21%); analysis of primary outcome 4/24 (17%); and numbers lost and excluded 4/24 (17%).

We conducted a comprehensive review of recently published SW-CRCT flow diagrams that, to our knowledge, has previously not been done before. Including a flow diagram when reporting SW-CRCTs is strongly recommended; however, when it came to reporting our SOCLE II trial we found that there were no suitable templates flexible enough to accommodate our SW-CRCT or comprehensive enough to support reporting all data items to standards. There was a wide variety in flow diagram quality, with the ‘best’ flow diagram only supporting the report of 13/22 (59%) of our required criteria. Additionally, due to the complexities of the diagrams they were often very difficult to interpret on their own without considering the full text.

A limitation of our review was that, although there are undoubtedly other SW-CRCT flow diagrams available, we chose to restrict our review to the 24 articles cited in Grayling et al (10) 2017 systematic review.

In the absence of a flow diagram template that considered all the items we required from the relevant reporting standards for our pilot SW-CRCT, we developed our own using Microsoft® Visio software (see Fig. 1 for the flow diagram populated with data from SOCLE II for illustrative purposes). Developing a suitable flow diagram for our study was a challenge requiring many iterations to ensure we were able to meet the required reporting standards. The development team included medical statisticians, a service user and health services researchers to ensure that what was represented in the flow diagram correlated with the statistical analyses and facilitated transparency and reporting quality.

Fig. 1 Flow Diagram from SOCLE II SW-CRCT – sites, staff & patient participants

Our unique flow diagram template was flexible enough to accommodate the reporting of our complex pragmatic SW-CRCT across multiple sites, dual participant groups, three study phases and the movement of participants across these. Various profiles are possible for our flow diagram that support selective filtering to enable focussed reporting at cluster or participant levels, by study phase, and movement of people across phases (see examples in Fig. 2 and Fig. 3). This degree of flexibility could not easily be facilitated in many of the other flow diagrams that we reviewed.

Fig. 2 Focussed flow diagram 1 – staff participants by site

Fig. 3 Focussed flow diagram 2 – overview summary of sites and participants

We believe that our flow diagram addresses the recommendations in the newly published CONSORT extension for SW-CRCTs, as it allows the reader to examine the nature of any differential inclusion and attrition by allocated sequence, treatment condition, and over time (5)(Hemming et al, 2018). An additional feature that could enhance our flow diagram would be to include time periods to allow for assessment of differential participation over time (5)(Hemming et al, 2018). We also recognise that with increasing numbers of clusters clear diagrammatic representation becomes increasingly difficult, and while our template can accommodate these larger numbers some summary reporting would be required for journal reports.

A flow diagram is a very powerful means of detailing a SW-CRCT design (10)(Grayling et al, 2017); therefore, researchers should ensure that prior to designing them they refer to the latest reporting recommendations in the new SW-CRCT CONSORT extension (5)(Hemming et al, 2018), along with any other relevant reporting standards and guidelines.

Our review of published SW-CRCT flow diagrams has highlighted that they currently provide inadequate support for the complete and transparent reporting of SW-CRCT data to required standards. It is unlikely that one flow diagram design will be applicable for all SW-CRCTs; however, our flow diagram template was used to good effect in our trial report and supported reporting to six different reporting standards. Our template is flexible enough to accommodate alternative SW-CRCT designs and provides important support to other triallists reporting their own SW-CRCTs to ensure that relevant data items are captured and reported.

Ethics approval and consent to participate

SOCLE II secured ethical approval from the Scotland A Research Ethics Committee Scotland (Reference number: 13/SS/0134). Informed consent to participate in the study was obtained from participants or their welfare guardians relatives, when required.

Consent for publication

Not applicable

Availability of data and material

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Competing interests

None of the authors have any competing interests in the manuscript. CJW was supported in this work by NHS Lothian via the Edinburgh Clinical Trials Unit.

Funding

Funding for the SOCLE II Trial was from the Stroke Association; however, there was no funding in the design of this current study and collection, analysis, and interpretation of data or in writing the manuscript.

Authors' contributions

MB conceived idea for paper, contributed to writing the manuscript and designed the SOCLE II flow diagram; HG was main author; BB and HG extracted data; BB contributed to writing the manuscript; CJW and CK provided input and feedback on drafts; JM contributed to the design of SOCLE II flow diagram.

Acknowledgements

Not applicable

(1) Moher D, Schulz K, Altman D. The CONSORT statement: Revised recommendations for improving the quality of reports on parallel-group randomised trials. The Lancet 2001;357(9263):1191-4.

(2) Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996;276(8):637-639.

(3) Campbell MK, Elbourne DR, Altman DG, CONSORT group. CONSORT statement: extension to cluster randomised trials. BMJ 2004 Mar 20;328(7441):702-708.

(4) Campbell MK, Piaggio G, Elbourne DR, Altman DG, CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ 2012 Sep 4;345:e5661.

(5) Hemming K, Taljaard M, McKenzie J, Hooper R, Copas A, Thompson J, et al. Reporting of The CONSORT extension for Stepped-Wedge Cluster Randomised Trials: Extension of the CONSORT 2010 statement with explanation and elaboration. British Medical Journal 2018.

(6) Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ : British Medical Journal 2015 Feb 6,;350(feb06 1):h391.

(7) Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006;185(5):263.

(8) Beard E, Lewis JJ, Copas A, Davey C, Osrin D, Baio G, et al. Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014. Trials 2015;16:353.

(9) Martin J, Taljaard M, Girling A, Hemming K. Systematic review finds major deficiencies in sample size methodology and reporting for stepped-wedge cluster randomised trials. BMJ Open 2016 Feb 4;6(2):010166.

(10) Grayling MJ, Wason JM, Mander AP. Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials 2017;18(1):33.

(11) Brady MC, Stott D, Weir CJ, Chalmers C, Sweeney P, Donaldson C, et al. Clinical and cost effectiveness of enhanced oral healthcare in stroke care settings (SOCLE II): A pilot, stepped wedge, cluster randomized, controlled trial protocol. International Journal of Stroke 2015;10(6):979-984.

(12) Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials 2010 Mar 24,;11(1):32.

(13) Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008;337(7680):1223-1226.

(14) Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ (Clinical research ed.) 2016 Oct 24,;355:i5239.

(15) Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. International journal of surgery 2014;12(12):1495-1499.

(16) Boutron I, Altman DG, Moher D, Schulz KF, Ravaud P. CONSORT statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts. Ann Intern Med 2017;167(1):40-47.

Download PDF

Version 1

posted

You are reading this latest preprint version

Rigorous Reporting of Stepped Wedge Cluster Randomised Controlled Trials: Development of a Flow Diagram Template

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and material

Funding

Authors' contributions

References

Supplementary Files

Status:

Version 1