Development of A Guideline for Reporting Mediation Analyses (AGReMA)

DOI: https://doi.org/10.21203/rs.2.18324/v1

Abstract

Background There are a growing number of studies using mediation analysis to understand the mechanisms of health interventions and exposures. Recent work has shown that the reporting of these studies is heterogenous and incomplete. This problem stifles clinical application, reproducibility, and evidence synthesis. This paper describes the processes and methods that will be used to develop a guideline for reporting studies of mediation analyses (AGReMA).

Methods/Design AGReMA will be developed over five overlapping stages. Stage one will comprise a systematic review to examine relevant evidence on the quality of reporting in published studies that use mediation analysis. In the second stage we will consult a group of methodologists and applied researchers by using a Delphi process to identify items that should be considered for inclusion in AGReMA. The third stage will involve a consensus meeting to consolidate and prioritise key items to be included in AGReMA. The fourth stage will involve production of AGReMA and an accompanying explanation and elaboration document. In the final stage we will disseminate the AGReMA statement via journals, conferences, and professional meetings across multiple disciplines.

Discussion The development and implementation of AGReMA will improve the standardization, transparency, and completeness in the reporting of studies that use mediation analysis to understand the mechanisms of health interventions and exposures.

BACKGROUND

The effects of exposures and health interventions are presumed to work via biological or psychosocial mechanisms. In recent years, epidemiologists and clinical trialists have used mediation analysis to understand the causal mechanisms by which exposures and interventions exert their effects on health outcomes [13]. The use of mediation analysis to understand the mechanisms of health interventions has been advocated by the US National Institute of Health (NIH), UK National Institute for Health Research (NIHR), and UK Medical Research Council (MRC) and has become increasingly common in recent years [1, 2].

While the application of mediation analysis is becoming popular, there is growing recognition that the reporting of studies investigating causal mechanisms of healthcare interventions is heterogenous and often incomplete [1, 410]. A recent overview of reviews across 11 health care fields and 26 health care conditions showed that mediation studies often did not report effect sizes and precision estimates, the theoretical rationale for the mechanism being tested, and essential details of the analytical techniques [1]. Reviews of the field show that the reporting of effect estimates and assumptions in primary mediation studies is varied across the literature [4] and most mediation analyses of randomised trials do not report sample size calculations [10]. These limitations stifle clinical application, reproducibility, and evidence synthesis. Published reporting guidelines such as the CONSORT [11], STROBE [12] and their extensions are not directly applicable, and there is no specific guidance for the reporting of studies that use mediation analysis.

To overcome the problem of suboptimal reporting, the US Berkeley Initiative for Transparency in the Social Sciences and the Center for Effective Global Action funded a project to develop a guideline for reporting mediation analyses. This paper describes the processes and methods that will be used to develop A Guideline for Reporting Mediation Analyses (AGReMA).

METHODS/DESIGN

AGReMA will be developed over five overlapping stages in accordance with the guidance for development of health research reporting guidelines [13] – Fig. 1:

1.
Systematic review to examine relevant evidence on the quality of reporting in published studies that use mediation analysis
2.
Delphi study to generate a consensus opinion on key items to be considered for inclusion in AGReMA
3.
Consensus meeting to consolidate and prioritise key items to be included in AGReMA and to structure an explanation and elaboration (E&E) document
4.
Write-up of the draft AGReMA statement and accompanying E&E document
5.
Dissemination and implementation of AGReMA

Stage 1 – systematic review

This study aims to systematically review the quality of reporting in published studies that use mediation analysis. Assessing the quality of reporting will provide important insights into the prevalence of potential sources of bias in studies that use mediation analysis and on reporting items to be considered for the eventual guideline [13]. Vo et al. 2019 has reviewed the reporting quality of randomised trials that use mediation analysis [10]. We will systematically review the reporting quality of non-randomised observational studies that have used mediation analysis to investigate causal mechanisms. The protocol for this systematic review was registered on the 29th May 2019 the International Prospective Register of Systematic Reviews (PROSPERO ID: CRD42019136348).

Databases and search terms

We will search EMBASE (OvidSP), MEDLINE (OvidSP) and PsycINFO (OvidSP) for non-randomised observational studies published in the previous two years that used mediation analysis. We will use the following search terms: mediation analysis, causal mediation, structural equation modelling, product of coefficient, indirect effect, direct effect, mechanism, intermediate variables [14]. We will not restrict our search based on health condition, journal, or type of exposure/intervention; mediator(s) and outcome(s) investigated to ensure a representative sample of up to 50 studies across healthcare are included. Our sample size was informed by previous systematic reviews that investigated the reporting quality of epidemiological studies [15, 16].

Data screening and selection

After removing duplicates, we will randomly order the identified records and select a random sample of up to 50 studies. Two reviewers will independently apply the inclusion/exclusion criteria (Table 1) sequentially to the random sample until perfect congruence is achieved between reviewers. Following this, one reviewer will screen the remaining studies independently until up to 50 studies have been included. Disagreements between reviewers at this stage will be discussed and resolved by consensus.

Table 1
Systematic review inclusion and exclusion criteria
Inclusion criteria
Design: Non-randomised observational studies (cohort, case-control, cross-sectional and non-randomized controlled trials) that used mediation analysis to understand causal mechanisms
 
Population: Individuals with any health condition, or individuals at risk of developing any health condition
 
Intervention or exposure: Any health intervention or exposure
 
Comparator: Any comparator
 
Outcomes: Any health-related outcome reported in the studies
Exclusion criteria
Reports of randomized controlled trials or non-randomised observational studies that have not used mediation analysis
 
Articles for which full texts are unavailable
 
Non-English-language articles

Data extraction

A single reviewer will independently extract data using a customised data extraction form. A second reviewer will verify the data extraction for ten percent of the included studies, with discrepancies to be resolved through discussion. First, we will extract descriptive information from each study including: authors, year of publication, journal, healthcare field; study design (cohort, case-control, cross-sectional and non-randomized controlled trials); publication type (primary or secondary publication using mediation analysis); aim of study; sample size; health condition; exposure/intervention; comparison group, outcomes measured; outcome measures; mediators investigated; mediators measures; time points measured. Secondly, we will extract specific information concerning the reporting quality of the methods and results of the mediation analysis. The assessment of reporting quality concerns the reporting of items identified from a scoping review of existing methodological and reporting guidance documents for mediation analyses, and from the findings of our recent overview of systematic reviews [1]. Disagreement during data extraction will be resolved through consensus and where necessary, by a third independent reviewer. Study data will be managed using REDCap electronic data capture tools [17].

Data synthesis

We will summarise the descriptive information using frequencies and percentages for categorical variables and mean and standard deviation or median and interquartile range for continuous variables. We will report the number and proportion of studies which report each of the prespecified reporting items.

Outcomes of systematic review

The results of this review will provide evidence on the quality of reporting of non-randomised observational studies that use mediation analysis. These findings will be used alongside existing systematic reviews [4, 9, 10] to inform the Delphi study (stage 2), and the consensus meeting (stage 3) to decide on the core reporting items for AGReMA. The findings of this review will be submitted for publication.

Stage 2 – Delphi Study

The aim of the Delphi study is to generate expert consensus on a list of items that should be reported in a mediation study. The process includes consulting experts to (1) assess the level of agreement on an initial list of reporting items generated from previous reviews; (2) elicit additional items and refine the initial list; and (3) identify which items are considered most important in reporting mediation studies to inform the consensus meeting.

Ethics

Ethics approval has been obtained from the University of New South Wales Human Research Ethics Advisory Panel D: Biomedical, approval number HC16599.

Design

The Delphi technique is a structured method to achieve consensus among a panel of experts on a given question or topic [18]. This process will comprise of a series of questionnaires or ‘rounds’, where panellists independently and anonymously contribute and rank items. This process is repeated for 3 rounds, or until consensus is reached. Following each round, panellists will be provided with summary feedback to encourage the reassessment of judgements for subsequent rounds, assisting in transforming individual opinion into group consensus [18].

Participants

We will invite experts who represent key stakeholders in the methodological development and application of mediation analysis to investigate causal mechanisms to be included in the Delphi panel. We will invite experts who have published original research papers involving mediation analysis or systematic reviews of mediation studies; methodological/statistical research papers on mediation analysis; or a textbook on mediation analysis. We aim to include between 7 to 15 participants in the Delphi study [19].

Recruitment process

Potential panellists will be identified through a variety of sources, including an overview [1] and a scoping review of the literature, and through consultation with experts. Recruitment will be iterative, with the final list of potential panellists decided through consensus amongst the AGReMA working group.

Procedure

CLINVIVO (www.clinvivo.com), an independent company will co-ordinate the web-based Delphi study to limit biases from the AGReMA working group [20].

Round 1

A questionnaire will be sent via email to the panellists. The questionnaire will include a statement about the purpose of project, demographic questions and reporting items for consideration. Panellists will be asked to score the importance of each potential reporting item on a 9-point Likert scale (1, ”not important”, to 9, “critically important”) and to describe their confidence in their ratings (1, “not confident”, to 9, “very confident”). Free text space will be provided at the end of each section to enable panellists to provide suggestions on wording. In addition, panellists will be asked to contribute additional items for consideration in subsequent rounds.

Round 2

Panellists who complete the first round will be sent a second-round questionnaire. This will include a summary of results from round one (mean scores and their standard deviations, median scores and inter-percentile ranges (IPR) (30th and 70th), histograms and Research ANd Development/University of California Los Angeles (RAND/UCLA) labels (see analysis below) of importance and agreement level), together with the panellist’s own score for each item. Newly nominated items and suggested re-wording of items from round 1 will also be presented. Panellists will be invited to re-score the importance of each item in the light of the aggregated panel medians. Panellists will be reminded that items scored ≤ 3 are considered not important and will be excluded and items scored ≥ 7 are considered critically important and will be included in the reporting guideline. Free text space will be provided at the end of each section to enable panellists to provide further suggestions on wording.

Round 3

Panellists who complete the second round will be sent a third-round questionnaire including a summary of results from round two (mean scores and their standard deviations, median scores and IPRs (30th and 70th), histograms and RAND/UCLA labels of importance and agreement level) for each item alongside the panellist’s own score. Panellists will be informed about the items that reached consensus for inclusion (median score ≥ 7) and exclusion (median score ≤ 3). Panellists will be asked to rate the remaining items for which consensus has not been reached (median score 4–6 or where disagreement exists) as: ‘Include’ or ‘Exclude’ in the final checklist. Panellist will also be asked to score their confidence in their ratings on a 9-point Likert Scale (1, “not confident”, to 9, “very confident”).

Analysis

The demographic information will be summarised with descriptive statistics. The free text comments from round 1 and 2 will be coded and thematically analysed to identify the key issues and common themes. This information will inform the re-wording of items and the addition of new items consideration in subsequent rounds.

We will use a modified version of the RAND/UCLA appropriateness method to analyse the responses from each round. We modified this approach by asking panellists to rate “importance” rather than “appropriateness”. The RAND/UCLA appropriateness method considers the median panel rating and dispersion of each panel rating to provide an index of appropriateness/importance and agreement [19]. This involves calculating the median score, the IPR (30th and 70th ) and the inter-percentile range adjusted for symmetry (IPRAS) for each item being rated. We will consider disagreement to be present in cases where IPR > IPRAS for a given item [19]. For the analysis of the round 1 and 2 responses, items to be considered for the reporting guideline will be categorised following the RAND/UCLA definitions [19]:

“Include”: panel median of 7–9 for importance, without disagreement
“Uncertain”: panel median of 4–6 for importance, or any median with disagreement
“Exclude”: panel median of 1–3 for importance, without disagreement

For the analysis of round 3, items to be considered for the reporting guideline will be categorised as [21]:

“Include”: panel majority as include
“Exclude”: panel majority as exclude

Outcome of Delphi Study

At the completion of this Delphi study, we will have reached consensus on a minimum sufficient list of items that should be considered for the reporting guideline in the consensus meeting (stage 3). The report of the Delphi study will be submitted for publication.

Stage 3 – Consensus meeting

A face-to-face consensus meeting [22] will be held to decide on the most important reporting items to be included in the AGReMA statement and to develop the accompanying explanation and elaboration document [13]. The consensus meeting will follow the methods suggested for developers of health research reporting guidelines [13].

Procedure

Participants will be recruited by the AGReMA working group to ensure the expertise of the participants is reflective of all relevant stakeholders (including trialists, epidemiologists, methodologists, statisticians, applied researchers and journal editors). Participants will be invited to attend a 1-day face-to-face consensus meeting. Prior to attending, the participants will be provided with the findings from the systematic review and the Delphi study. The meeting will include presentations of the evidence for reporting quality of mediation studies, and results of the Delphi study. A member of the AGReMA working group will facilitate a structured discussion on the rationale of including each item identified in the Delphi study. Participants will be given opportunity to discuss each item. In cases of disagreement, an anonymised vote will be held to establish prioritisation of the item for inclusion in AGReMA. The meeting will conclude with discussion about the content and production of relevant documents (AGReMA statement, E&E paper, etc.) as well as strategies to optimise dissemination and implementation. Following the conclusion of the meeting, a written report on the meeting outcome will be circulated to the consensus meeting participants for comment and approval.

Stage 4 Development of the draft AGReMA statement and E&E document

The purpose of this stage is to draft the statement and accompanying E&E document to ensure that wording and content is clear, precise, and suitable for all relevant stakeholders.

The purpose of the E&E document is to describe the background, rationale and justification for each reporting item and provide an example of clear reporting for each item. This is designed to help clarify the importance of each item, highlight relevant reporting issues, and assist authors in meeting the AGReMA statement requirements. The expert consensus meeting participants will be consulted to review and comment on the draft documents.

Stage 5 – Guideline dissemination

The goal of the final stage is to maximise the awareness, accessibility, and utilisation of AGReMA. The dissemination strategy will be informed and guided by the AGReMA working group and consensus meeting participants. We aim to produce simultaneous publications in several high-reach journals to begin the process of dissemination and uptake, accompanied by a social media dissemination strategy. We will liaise with relevant journal editors and funding agencies to encourage AGReMA endorsement alongside other reporting guidelines eg. CONSORT, PRISMA etc. In addition to open-access publications, we will make the AGReMA statement and its E&E document available on an open AGReMA web-domain, and index it on the EQUATOR Network website and Penelope.ai [23]. We will create a suite of online resources including audio-visual guides which will be available on the AGReMA web-domain to assist application. The AGReMA working group will disseminate the statement at relevant conferences and statistical/methodological courses. Finally, the AGReMA statement and accompanying resources will shared directly with authors that routinely use mediation analysis.

Publication plan

Publication 1: Study protocol
Publication 2: Systematic review
Publication 3: Delphi study
Publication 4 & 5: simultaneous publications for the AGReMA statement and E&E paper

DISCUSSION

The number of studies using mediation analysis to understand the mechanisms of health exposures and interventions is increasing [1, 2]. However, the reporting of these studies remains heterogenous and incomplete [1, 410]. Methods synthesising mediation studies are also under development so a reporting guideline is timely to help reduce the heterogeneity and facilitate the synthesis and pooling of mediation studies. Although the application of mediation analyses is relatively new, early development of a standardised reporting guideline will prevent misreported studies and subsequent research waste [24]. A reporting guideline would not only assist in the production and review of research manuscripts, but also in the appraisal and implementation of the study findings by researchers, clinicians, patients, funders and policy makers.

This project will produce a reporting guideline for studies that use mediation analysis to investigate causal mechanisms in healthcare research. To ensure this guideline is useful and widely used, it is being developed using comprehensive, robust and widely accepted methods [13]. We will also use a structured dissemination strategy to ensure implementation and uptake of AGReMA. This strategy will be targeted at all relevant stakeholders. We will ensure that the guideline is both available and usable and we will develop a suite of resources to support guideline use.

Declarations

Ethics approval and consent to participate

Ethics approval was obtained from the University of New South Wales Human Research Ethics Advisory Panel D: Biomedical, approval number HC16599.

 

Consent for publication

Not applicable

 

Availability of data and materials

Not applicable

 

Competing interests

AC is supported by the University of New South Wales Prince of Wales Clinical School Postgraduate Research Scholarship and a NeuRA PhD Candidature Supplementary Scholarship, and is a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences. HL is funded by the National Health and Medical Research Council (grant no. APP1126767); National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care Oxford at Oxford Health NHS Foundation Trust; received project funding from the Berkeley Initiative for Transparency in the Social Sciences, a program of the Center for Effective Global Action (CEGA), with support from the Laura and John Arnold Foundation; and is a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences. SK is funded by the National Health and Medical Research Council (APP1127932).

 

Funding

This work was supported by project funding from the Berkeley Initiative for Transparency in the Social Sciences, a program of the Center for Effective Global Action (CEGA), with support from the Laura and John Arnold Foundation. The funding body did not contribute to the design of the study, the collection, analysis and interpretation of data and in writing the manuscript.

 

Authors’ contributions

AC, HL and JM conceived the idea for the project. AC and HL wrote the first draft of the manuscript. All authors provided feedback on the manuscript and have read and approved the final version.

 

Acknowledgments

None

 

 

FIGURE TITLE

Figure 1. Workflow for AGReMA: A Guideline for Reporting Mediation Analyses

 

References

  1. Cashin AG, Lee H, Lamb SE, Hopewell S, Mansell G, Williams CM, et al. An overview of systematic reviews found suboptimal reporting and methodological limitations of mediation studies investigating causal mechanisms. J Clin Epidemiol. 2019;111:60-68. https://doi.org/10.1016/j.jclinepi.2019.03.005
  2. Nguyen TQ, Schmid I, Stuart EA. Clarifying causal mediation analysis for the applied researcher: Defining effects based on what we want to learn. 2019; Available from: http://arxiv.org/abs/1904.08515
  3. Lee H, Herbert RD, Lamb SE, Moseley AM, McAuley JH. Investigating causal mechanisms in randomised controlled trials. Trials. 2019;20:524. https://doi.org/10.1186/s13063-019-3593-z
  4. Liu S, Ulbricht CM, Chrysanthopoulou SA, Lapane KL. Implementation and reporting of causal mediation analysis in 2015: a systematic review in epidemiological studies. BMC Res Notes. 2016;9. https://doi.org/10.1186/s13104-016-2163-7
  5. Gelfand LA, Mensinger JL, Tenhave T. Mediation Analysis: A Retrospective Snapshot of Practice and More Recent Directions NIH Public Access. J Gen Psychol. 2009;136:153–76. http://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC2670477&blobtype=pdf
  6. Mansell G, Kamper SJ, Kent P. Why and how back pain interventions work: What can we do to find out? Best Pract Res Clin Rheumatol. 2013;27:685–97.
  7. Hertzog M. Trends in Mediation Analysis in Nursing Research: Improving Current Practice. West J Nurs Res. 2018;40:907–30. https://doi.org/10.1177/0193945916687782
  8. Wood RE, Goodman JS, Beckmann N, Cook A. Mediation testing in management research: A review and proposals. Organ Res Methods. 2008;11:270–95.
  9. Lapointe-Shaw L, Bouck Z, Howell NA, Lange T, Orchanian-Cheff A, Austin PC, et al. Mediation analysis with a time-to-event outcome: a review of use and reporting in healthcare research. BMC Med Res Methodol. 2018;18:118. https://doi.org/10.1186/s12874-018-0578-7
  10. Vo T, Superchi C, Boutron I, Vansteelandt S. The conduct and reporting of mediation analysis in recently published randomized controlled trials: results from a methodological systematic review. J Clin Epidemiol; 2019. https://doi.org/10.1016/j.jclinepi.2019.10.001
  11. Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:18. http://www.biomedcentral.com/1741-7015/8/18
  12. Elm E von, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335:806–8. http://www.ncbi.nlm.nih.gov/pubmed/17947786
  13. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;e1000217. http://dx.plos.org/10.1371/journal.pmed.1000217
  14. Lee H, Hübscher M, Moseley GL, Kamper SJ, Traeger AC, Mansell G, et al. How does pain lead to disability? A systematic review and meta-analysis of mediation studies in people with back and neck pain. Pain. 2015;156:988–97.
  15. Phillips AC, Lewis LK, McEvoy MP, Galipeau J, Glasziou P, Hammick M, et al. A systematic review of how studies describe educational interventions for evidence-based practice: stage 1 of the development of a reporting guideline. BMC Med Educ. 2014;14:152. https://bmcmededuc.biomedcentral.com/articles/10.1186/1472-6920-14-152
  16. Tooth L, Ware R, Bain C, Purdie DM, Dobson A. Quality of Reporting of Observational Longitudinal Research. Am J Epidemiol. 2005;161:280–8. http://www.ncbi.nlm.nih.gov/pubmed/15671260
  17. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81. http://www.ncbi.nlm.nih.gov/pubmed/18929686
  18. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–80. http://www.ncbi.nlm.nih.gov/pubmed/7640549
  19. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, Lacalle JR, Lázaro P, et al. The RAND/UCLA Appropriateness Method User’s Manual Library of Congress Cataloging-in-Publication Data. 2001
  20. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31:684–706. https://doi.org/10.1177/0269216317690685
  21. Froud R, Eldridge S, Kovacs F, Breen A, Bolton J, Dunn K, et al. Reporting outcomes of back pain trials: A modified Delphi study. Eur J Pain. 2011;15:1068–74. https://onlinelibrary.wiley.com/doi/pdf/10.1016/j.ejpain.2011.04.015
  22. Sanderson CFB, Askham J, Adam S, Infirmary R, Russell I, James JH, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2:1–83.
  23. Equator Network. equator-wizard — Penelope.ai [Internet]. 2016. https://www.penelope.ai/equatorwizard
  24. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–9. https://doi.org/10.1016/S0140-6736(09)60329-9