DOI: https://doi.org/10.21203/rs.2.18324/v1
Background There are a growing number of studies using mediation analysis to understand the mechanisms of health interventions and exposures. Recent work has shown that the reporting of these studies is heterogenous and incomplete. This problem stifles clinical application, reproducibility, and evidence synthesis. This paper describes the processes and methods that will be used to develop a guideline for reporting studies of mediation analyses (AGReMA).
Methods/Design AGReMA will be developed over five overlapping stages. Stage one will comprise a systematic review to examine relevant evidence on the quality of reporting in published studies that use mediation analysis. In the second stage we will consult a group of methodologists and applied researchers by using a Delphi process to identify items that should be considered for inclusion in AGReMA. The third stage will involve a consensus meeting to consolidate and prioritise key items to be included in AGReMA. The fourth stage will involve production of AGReMA and an accompanying explanation and elaboration document. In the final stage we will disseminate the AGReMA statement via journals, conferences, and professional meetings across multiple disciplines.
Discussion The development and implementation of AGReMA will improve the standardization, transparency, and completeness in the reporting of studies that use mediation analysis to understand the mechanisms of health interventions and exposures.
The effects of exposures and health interventions are presumed to work via biological or psychosocial mechanisms. In recent years, epidemiologists and clinical trialists have used mediation analysis to understand the causal mechanisms by which exposures and interventions exert their effects on health outcomes [1–3]. The use of mediation analysis to understand the mechanisms of health interventions has been advocated by the US National Institute of Health (NIH), UK National Institute for Health Research (NIHR), and UK Medical Research Council (MRC) and has become increasingly common in recent years [1, 2].
While the application of mediation analysis is becoming popular, there is growing recognition that the reporting of studies investigating causal mechanisms of healthcare interventions is heterogenous and often incomplete [1, 4–10]. A recent overview of reviews across 11 health care fields and 26 health care conditions showed that mediation studies often did not report effect sizes and precision estimates, the theoretical rationale for the mechanism being tested, and essential details of the analytical techniques [1]. Reviews of the field show that the reporting of effect estimates and assumptions in primary mediation studies is varied across the literature [4] and most mediation analyses of randomised trials do not report sample size calculations [10]. These limitations stifle clinical application, reproducibility, and evidence synthesis. Published reporting guidelines such as the CONSORT [11], STROBE [12] and their extensions are not directly applicable, and there is no specific guidance for the reporting of studies that use mediation analysis.
To overcome the problem of suboptimal reporting, the US Berkeley Initiative for Transparency in the Social Sciences and the Center for Effective Global Action funded a project to develop a guideline for reporting mediation analyses. This paper describes the processes and methods that will be used to develop A Guideline for Reporting Mediation Analyses (AGReMA).
AGReMA will be developed over five overlapping stages in accordance with the guidance for development of health research reporting guidelines [13] – Fig. 1:
This study aims to systematically review the quality of reporting in published studies that use mediation analysis. Assessing the quality of reporting will provide important insights into the prevalence of potential sources of bias in studies that use mediation analysis and on reporting items to be considered for the eventual guideline [13]. Vo et al. 2019 has reviewed the reporting quality of randomised trials that use mediation analysis [10]. We will systematically review the reporting quality of non-randomised observational studies that have used mediation analysis to investigate causal mechanisms. The protocol for this systematic review was registered on the 29th May 2019 the International Prospective Register of Systematic Reviews (PROSPERO ID: CRD42019136348).
We will search EMBASE (OvidSP), MEDLINE (OvidSP) and PsycINFO (OvidSP) for non-randomised observational studies published in the previous two years that used mediation analysis. We will use the following search terms: mediation analysis, causal mediation, structural equation modelling, product of coefficient, indirect effect, direct effect, mechanism, intermediate variables [14]. We will not restrict our search based on health condition, journal, or type of exposure/intervention; mediator(s) and outcome(s) investigated to ensure a representative sample of up to 50 studies across healthcare are included. Our sample size was informed by previous systematic reviews that investigated the reporting quality of epidemiological studies [15, 16].
After removing duplicates, we will randomly order the identified records and select a random sample of up to 50 studies. Two reviewers will independently apply the inclusion/exclusion criteria (Table 1) sequentially to the random sample until perfect congruence is achieved between reviewers. Following this, one reviewer will screen the remaining studies independently until up to 50 studies have been included. Disagreements between reviewers at this stage will be discussed and resolved by consensus.
Inclusion criteria | Design: Non-randomised observational studies (cohort, case-control, cross-sectional and non-randomized controlled trials) that used mediation analysis to understand causal mechanisms |
---|---|
Population: Individuals with any health condition, or individuals at risk of developing any health condition | |
Intervention or exposure: Any health intervention or exposure | |
Comparator: Any comparator | |
Outcomes: Any health-related outcome reported in the studies | |
Exclusion criteria | Reports of randomized controlled trials or non-randomised observational studies that have not used mediation analysis |
Articles for which full texts are unavailable | |
Non-English-language articles |
A single reviewer will independently extract data using a customised data extraction form. A second reviewer will verify the data extraction for ten percent of the included studies, with discrepancies to be resolved through discussion. First, we will extract descriptive information from each study including: authors, year of publication, journal, healthcare field; study design (cohort, case-control, cross-sectional and non-randomized controlled trials); publication type (primary or secondary publication using mediation analysis); aim of study; sample size; health condition; exposure/intervention; comparison group, outcomes measured; outcome measures; mediators investigated; mediators measures; time points measured. Secondly, we will extract specific information concerning the reporting quality of the methods and results of the mediation analysis. The assessment of reporting quality concerns the reporting of items identified from a scoping review of existing methodological and reporting guidance documents for mediation analyses, and from the findings of our recent overview of systematic reviews [1]. Disagreement during data extraction will be resolved through consensus and where necessary, by a third independent reviewer. Study data will be managed using REDCap electronic data capture tools [17].
We will summarise the descriptive information using frequencies and percentages for categorical variables and mean and standard deviation or median and interquartile range for continuous variables. We will report the number and proportion of studies which report each of the prespecified reporting items.
The results of this review will provide evidence on the quality of reporting of non-randomised observational studies that use mediation analysis. These findings will be used alongside existing systematic reviews [4, 9, 10] to inform the Delphi study (stage 2), and the consensus meeting (stage 3) to decide on the core reporting items for AGReMA. The findings of this review will be submitted for publication.
The aim of the Delphi study is to generate expert consensus on a list of items that should be reported in a mediation study. The process includes consulting experts to (1) assess the level of agreement on an initial list of reporting items generated from previous reviews; (2) elicit additional items and refine the initial list; and (3) identify which items are considered most important in reporting mediation studies to inform the consensus meeting.
Ethics approval has been obtained from the University of New South Wales Human Research Ethics Advisory Panel D: Biomedical, approval number HC16599.
The Delphi technique is a structured method to achieve consensus among a panel of experts on a given question or topic [18]. This process will comprise of a series of questionnaires or ‘rounds’, where panellists independently and anonymously contribute and rank items. This process is repeated for 3 rounds, or until consensus is reached. Following each round, panellists will be provided with summary feedback to encourage the reassessment of judgements for subsequent rounds, assisting in transforming individual opinion into group consensus [18].
We will invite experts who represent key stakeholders in the methodological development and application of mediation analysis to investigate causal mechanisms to be included in the Delphi panel. We will invite experts who have published original research papers involving mediation analysis or systematic reviews of mediation studies; methodological/statistical research papers on mediation analysis; or a textbook on mediation analysis. We aim to include between 7 to 15 participants in the Delphi study [19].
Potential panellists will be identified through a variety of sources, including an overview [1] and a scoping review of the literature, and through consultation with experts. Recruitment will be iterative, with the final list of potential panellists decided through consensus amongst the AGReMA working group.
CLINVIVO (www.clinvivo.com), an independent company will co-ordinate the web-based Delphi study to limit biases from the AGReMA working group [20].
A questionnaire will be sent via email to the panellists. The questionnaire will include a statement about the purpose of project, demographic questions and reporting items for consideration. Panellists will be asked to score the importance of each potential reporting item on a 9-point Likert scale (1, ”not important”, to 9, “critically important”) and to describe their confidence in their ratings (1, “not confident”, to 9, “very confident”). Free text space will be provided at the end of each section to enable panellists to provide suggestions on wording. In addition, panellists will be asked to contribute additional items for consideration in subsequent rounds.
Panellists who complete the first round will be sent a second-round questionnaire. This will include a summary of results from round one (mean scores and their standard deviations, median scores and inter-percentile ranges (IPR) (30th and 70th), histograms and Research ANd Development/University of California Los Angeles (RAND/UCLA) labels (see analysis below) of importance and agreement level), together with the panellist’s own score for each item. Newly nominated items and suggested re-wording of items from round 1 will also be presented. Panellists will be invited to re-score the importance of each item in the light of the aggregated panel medians. Panellists will be reminded that items scored ≤ 3 are considered not important and will be excluded and items scored ≥ 7 are considered critically important and will be included in the reporting guideline. Free text space will be provided at the end of each section to enable panellists to provide further suggestions on wording.
Panellists who complete the second round will be sent a third-round questionnaire including a summary of results from round two (mean scores and their standard deviations, median scores and IPRs (30th and 70th), histograms and RAND/UCLA labels of importance and agreement level) for each item alongside the panellist’s own score. Panellists will be informed about the items that reached consensus for inclusion (median score ≥ 7) and exclusion (median score ≤ 3). Panellists will be asked to rate the remaining items for which consensus has not been reached (median score 4–6 or where disagreement exists) as: ‘Include’ or ‘Exclude’ in the final checklist. Panellist will also be asked to score their confidence in their ratings on a 9-point Likert Scale (1, “not confident”, to 9, “very confident”).
The demographic information will be summarised with descriptive statistics. The free text comments from round 1 and 2 will be coded and thematically analysed to identify the key issues and common themes. This information will inform the re-wording of items and the addition of new items consideration in subsequent rounds.
We will use a modified version of the RAND/UCLA appropriateness method to analyse the responses from each round. We modified this approach by asking panellists to rate “importance” rather than “appropriateness”. The RAND/UCLA appropriateness method considers the median panel rating and dispersion of each panel rating to provide an index of appropriateness/importance and agreement [19]. This involves calculating the median score, the IPR (30th and 70th ) and the inter-percentile range adjusted for symmetry (IPRAS) for each item being rated. We will consider disagreement to be present in cases where IPR > IPRAS for a given item [19]. For the analysis of the round 1 and 2 responses, items to be considered for the reporting guideline will be categorised following the RAND/UCLA definitions [19]:
For the analysis of round 3, items to be considered for the reporting guideline will be categorised as [21]:
At the completion of this Delphi study, we will have reached consensus on a minimum sufficient list of items that should be considered for the reporting guideline in the consensus meeting (stage 3). The report of the Delphi study will be submitted for publication.
A face-to-face consensus meeting [22] will be held to decide on the most important reporting items to be included in the AGReMA statement and to develop the accompanying explanation and elaboration document [13]. The consensus meeting will follow the methods suggested for developers of health research reporting guidelines [13].
Participants will be recruited by the AGReMA working group to ensure the expertise of the participants is reflective of all relevant stakeholders (including trialists, epidemiologists, methodologists, statisticians, applied researchers and journal editors). Participants will be invited to attend a 1-day face-to-face consensus meeting. Prior to attending, the participants will be provided with the findings from the systematic review and the Delphi study. The meeting will include presentations of the evidence for reporting quality of mediation studies, and results of the Delphi study. A member of the AGReMA working group will facilitate a structured discussion on the rationale of including each item identified in the Delphi study. Participants will be given opportunity to discuss each item. In cases of disagreement, an anonymised vote will be held to establish prioritisation of the item for inclusion in AGReMA. The meeting will conclude with discussion about the content and production of relevant documents (AGReMA statement, E&E paper, etc.) as well as strategies to optimise dissemination and implementation. Following the conclusion of the meeting, a written report on the meeting outcome will be circulated to the consensus meeting participants for comment and approval.
The purpose of this stage is to draft the statement and accompanying E&E document to ensure that wording and content is clear, precise, and suitable for all relevant stakeholders.
The purpose of the E&E document is to describe the background, rationale and justification for each reporting item and provide an example of clear reporting for each item. This is designed to help clarify the importance of each item, highlight relevant reporting issues, and assist authors in meeting the AGReMA statement requirements. The expert consensus meeting participants will be consulted to review and comment on the draft documents.
The goal of the final stage is to maximise the awareness, accessibility, and utilisation of AGReMA. The dissemination strategy will be informed and guided by the AGReMA working group and consensus meeting participants. We aim to produce simultaneous publications in several high-reach journals to begin the process of dissemination and uptake, accompanied by a social media dissemination strategy. We will liaise with relevant journal editors and funding agencies to encourage AGReMA endorsement alongside other reporting guidelines eg. CONSORT, PRISMA etc. In addition to open-access publications, we will make the AGReMA statement and its E&E document available on an open AGReMA web-domain, and index it on the EQUATOR Network website and Penelope.ai [23]. We will create a suite of online resources including audio-visual guides which will be available on the AGReMA web-domain to assist application. The AGReMA working group will disseminate the statement at relevant conferences and statistical/methodological courses. Finally, the AGReMA statement and accompanying resources will shared directly with authors that routinely use mediation analysis.
The number of studies using mediation analysis to understand the mechanisms of health exposures and interventions is increasing [1, 2]. However, the reporting of these studies remains heterogenous and incomplete [1, 4–10]. Methods synthesising mediation studies are also under development so a reporting guideline is timely to help reduce the heterogeneity and facilitate the synthesis and pooling of mediation studies. Although the application of mediation analyses is relatively new, early development of a standardised reporting guideline will prevent misreported studies and subsequent research waste [24]. A reporting guideline would not only assist in the production and review of research manuscripts, but also in the appraisal and implementation of the study findings by researchers, clinicians, patients, funders and policy makers.
This project will produce a reporting guideline for studies that use mediation analysis to investigate causal mechanisms in healthcare research. To ensure this guideline is useful and widely used, it is being developed using comprehensive, robust and widely accepted methods [13]. We will also use a structured dissemination strategy to ensure implementation and uptake of AGReMA. This strategy will be targeted at all relevant stakeholders. We will ensure that the guideline is both available and usable and we will develop a suite of resources to support guideline use.
Ethics approval and consent to participate
Ethics approval was obtained from the University of New South Wales Human Research Ethics Advisory Panel D: Biomedical, approval number HC16599.
Consent for publication
Not applicable
Availability of data and materials
Not applicable
Competing interests
AC is supported by the University of New South Wales Prince of Wales Clinical School Postgraduate Research Scholarship and a NeuRA PhD Candidature Supplementary Scholarship, and is a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences. HL is funded by the National Health and Medical Research Council (grant no. APP1126767); National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care Oxford at Oxford Health NHS Foundation Trust; received project funding from the Berkeley Initiative for Transparency in the Social Sciences, a program of the Center for Effective Global Action (CEGA), with support from the Laura and John Arnold Foundation; and is a Catalyst for the Berkeley Initiative for Transparency in the Social Sciences. SK is funded by the National Health and Medical Research Council (APP1127932).
Funding
This work was supported by project funding from the Berkeley Initiative for Transparency in the Social Sciences, a program of the Center for Effective Global Action (CEGA), with support from the Laura and John Arnold Foundation. The funding body did not contribute to the design of the study, the collection, analysis and interpretation of data and in writing the manuscript.
Authors’ contributions
AC, HL and JM conceived the idea for the project. AC and HL wrote the first draft of the manuscript. All authors provided feedback on the manuscript and have read and approved the final version.
Acknowledgments
None
FIGURE TITLE
Figure 1. Workflow for AGReMA: A Guideline for Reporting Mediation Analyses