3.1. Study design
This is a methods study in the knowledge synthesis field. We followed SR guidance for the study selection and data extraction stages [26]. Our protocol will be registered and located on the Open Science Framework at https://osf.io/nbcta/.
3.2 Search and selection of Discordant Reviews
3.2.1 Database of 1218 overviews
We will use a database of 1218 overviews published between January 1, 2000 and December 30, 2020 created from a bibliometric study [27] as the basis of our search for Discordant Reviews using Jadad or another method to assess discordance. A validated search filter for overviews [28] was used in MEDLINE (Ovid), Epistemonikos, and the Cochrane Database of Systematic Reviews (CDSR) for overviews. In an empirical methods study of the retrieval sensitivity of 6 databases, the combination of MEDLINE and Epistemonikos retrieved 95.2% of all systematic reviews [29]. As a rationale, we believe this combination would retrieve an equal proportion of overviews. The database Epistemonikos contains both published and unpublished reports. We searched the CDSR through the website interface using the filter for Cochrane reviews. Overviews included in the bibliometric study contain these characteristics: (a) synthesised the results of SRs, (b) systematically searched for evidence in a minimum of two databases, and (c) conducted a search using a combination of text words and MeSH terms. The included overviews also had to have a methods section in the main body of the paper and focused on health interventions. To identify studies assessing discordance using the database of 1218 overviews, we will use the EndNote search function and Boolean logic to search for the following words: overlap*[title/abstract] or discrepan*[title/abstract] or discord*[title/abstract] or concord*[title/abstract] or conflict*[title/abstract] or Jadad [abstract].
3.2.2 Medline (Ovid) search strategy January to April 2021
We will update this search with an Ovid MEDLINE search using the following search string: ("systematic reviews".ti,ab. or “meta-analyses”.ti,ab.) AND (overlap.ti,ab or discrepant.ti,ab or discordant.ti,ab. or difference.ti,ab. or conflicting.ti,ab. or Jadad.ab.). Our search was conducted on April 18, 2021.
3.3 Screening Discordant Reviews
3.3.1 Process for screening Discordant Reviews
Citations identified by our searches will be assigned a random number and screened sequentially. The first 30 Discordant Reviews screened at full-text and meeting our eligibility criteria will be included.
All authors will pilot the screening form on 20 Discordant Reviews to ensure high levels of agreement and common definitions of eligibility criteria.
Two authors will independently screen Discordant Reviews as full-text publications. Discrepancies will be resolved by consensus, and arbitration by a third reviewer when necessary.
3.3.2 Stage 1 screening criteria
We first include all reports aiming to assess discordant results across SRs on the same question. Studies assessing discordance can assess (a) discordant results, or (b) discordant interpretations of the results and conclusions. Both studies examining (a) and (b) were eligible (e.g. [30-33]).
If a study meets stage 1 criteria, it will be included in Study 1.
3.3.3 Stage 2 screening criteria
From this sample, we will then screen Discordant Reviews based on the following inclusion criteria:
- Included a minimum of two SRs with a meta-analysis of randomised controlled trials (RCTs), but may have included other study types beyond RCTs; and
- Used the Jadad algorithm.
If a study meets stage 2 screening, it will move onto stage 3 screening.
3.3.4 Stage 3 screening
After stage 2 screening is complete, we will screen based on the authorship team. When the same ‘core’ authors (first, last, and/or corresponding) conduct two or more of the identified Discordant Reviews, we will only include one of the multiple Discordant Reviews (i.e. the most recent study will be selected). Our rationale is that author groups use the same methods to assess discordance (e.g. Mascarenas [24] and Chalmers [15]).
We will include Discordant Reviews in any language and publication status, published anytime. We will use Google translate to interpret non-English studies for screening and assessment. The resulting publications will form the set for study 2.
In the case where the authors of this study are also authors of one or several of the included Discordant Reviews, those authors will not conduct screening, extraction, Jadad assessment, or analysis of the Discordant Review in question.
Full-texts of all SRs included in an Discordant Review will be obtained.
3.4 Extraction of the primary intervention and outcome
3.4.1 Identification of the primary outcome from the Discordant Review
As a first step in assessing discordance, we will identify the primary outcome from each Discordant Review. The primary outcome will be extracted when it is explicitly defined in the title, abstract, objectives, introduction or methods sections [34, 35]. If the primary outcome is not found in any of these sections, we will extract it as the first outcome mentioned in the manuscript [34, 35]. If the primary outcome cannot be identified by any of these approaches, we will consider that the article did not specify primary outcomes and the study will be excluded, and replaced with the next Discordant Review in our database.
3.4.2 Identification of the primary intervention from the Discordant Review
Identification of the primary outcome and intervention is a two-step process. As a first step, we will identify the primary intervention associated with the primary outcome from each Discordant Review. Then we will extract the primary outcome and intervention from the included systematic reviews when doing the Jadad assessments. If this is unclear, we will choose the first intervention highlighted in the title or abstract of the Discordant Review [35].
3.4.3 Identification of included SRs with meta-analysis of RCTs addressing the primary intervention/outcome of the Discordant Review
Once the primary outcome is identified, we will next identify how many SRs with meta-analysis of RCTs were included in the Discordant Review that address the primary outcome and intervention. It is this sample of SRs with meta-analysis of RCTs that will be the focus of our Jadad assessments.
3.4.4 Process to identify primary intervention and outcome
Two authors will extract the primary intervention and outcome, and disagreements will be discussed until consensus is reached.
3.5 Blinding of results in the included SRs
Observer bias, sometimes called “detection bias” or “ascertainment bias,” occurs when outcome assessments are systematically influenced by the assessors’ conscious or unconscious predispositions [36]. Blinded outcome assessors are used in trials to avoid such bias. One empirical study found evidence of high risk of substantial bias when authors failed to blind outcome assessors in trials [36], whereas another did not [37]. In our study, it is important that reviewers are blinded to the Discordant Review authors' result of the Jadad assessment, as unblinding might predispose them to unconsciously choose the same SR as the Discordant Review authors.
We will blind the following components containing study results of the Jadad assessment and conclusions: abstract, highlights, results of the Jadad assessment, and discussion/conclusions section. Blinding will be achieved via deletion using the paywalled Adobe Acrobat Pro or the freeware PDFCandy (https://pdfcandy.com). One author will blind the Discordant Review results and will not be involved in the Jadad assessment pertaining to those results. Assessors will be instructed not to search for and read the included Discordant Reviews prior to, and during, assessment.
3.6 Piloting Jadad assessment prior to full assessment
A pilot practice exercise will be conducted by all assessors prior to the Jadad assessments, to ensure consistent assessments across reviewers. Two Jadad assessments will be piloted by each reviewer and compared to a second to identify discrepancies that are to be resolved through discussion. Any necessary revisions to the assessment (sections 3.7 and 3.8) will be noted.
3.7 Jadad assessments of discordance across SRs
While the Jadad paper provides an algorithm intended to identify and address discordance between SRs in an Discordant Review, there is limited guidance within the manuscript regarding the application/operationalization of the algorithm. Absence of this detailed guidance leaves room for subjective (mis)interpretation and ultimately confusion when it comes time to use the algorithm. To address this, we engaged in an iterative process of interpretation and implementation of the algorithm step by step. This process involved virtual meetings whereby consensus was sought for decision rules at each step of the algorithm to ensure consistency in both interpretation and application. Feedback was solicited and decision rules were accordingly adjusted until consensus was achieved. This tool underwent pilot testing as described in 3.6 where feedback was further solicited and adjustments were made.
The Jadad decision tree assesses and compares sources of inconsistency between SRs with meta-analyses, including differences in clinical, public health, or policy questions, inclusion and exclusion criteria, extracted data, methodological quality assessments, data combining, and statistical analysis methods.
Step A is to examine the multiple SRs matching the Discordant Review question using a PICO framework. If the clinical, public health, or policy questions are not identical, then step B indicates choosing the SR closest to the decision makers’ research question and no further assessment is necessary. If multiple SRs are found with the same PICO, then step C should be investigated. As we are using Discordant Reviews as our sample, we will start at Step C in the Jadad decision tree.
Figure 1. Jadad (1997) decision tree.
Here we detail our interpretation of the Jadad algorithm for each step in assessing the discordance in a group of SRs with similar PICO elements. Two researchers will independently assess each set of SRs in the included Discordant Review using the Jadad algorithm, starting with Step C (Figure 1). Information and data from the Discordant Review will be used first if reported, and when data is not reported, we will consult the full text of the included SRs. If a Discordant Review or the included SRs does not report a method, we will indicate it as “not reported,” and it will not be chosen for that step.
Step D and G follow from Step C. Steps E, F, H. and I are completed depending on the decisions at Steps D and G, respectively.
“Meeting” a step means a SR met the criteria in the sub-step or step that is highest in the hierarchy. For example, a SR that meets E3 criteria fulfills criteria A and B, which is the highest in our hierarchy.
3.7.2 Step C - Do the reviews include the same trials for the primary intervention and outcome?
We will determine if the RCTs are similar across SRs by either finding this information in the Discordant Review, or extracting all RCTs from the included SRs using an excel matrix to list the SRs at the top, and trials in the left rows. The RCTs will be mapped to the SRs in order of publication date (earliest trials at the top). Using this matrix, we will determine if the SRs include the same or different trials.
3.7.3 Step D - Are the reviews of the same quality?
If the SRs contain the same trials, then the assessor moves to Step D – assess whether the SRs are the same methodological quality. We will either (a) extract the risk of bias/quality assessments from the Discordant Review if the Discordant Review used AMSTAR [38], AMSTAR 2 [39], or ROBIS [40]. AMSTAR [38] and the updated AMSTAR 2 [39] are tools to assess methodological quality (i.e. quality of conduct and reporting) and ROBIS [40] is a tool used to assess the risk of bias at the systematic-review level. Review-level biases include selective outcome reporting (e.g. only describing the statistically significant, and not describing all outcomes) and publication bias (e.g. published studies are more likely to report positive results). If the Discordant Review authors used any other tool or method to assess the risk of bias/quality of the SRs or did not assess the risk of bias/quality assessment at all, we will conduct our own assessment using the ROBIS tool [40].
3.7.4 Step E – Assess and compare data extraction, clinical heterogeneity, and data synthesis
If the SRs are the same risk of bias/quality, then the next step is Step E, to assess and compare data extraction, clinical heterogeneity, and data synthesis across the SRs.
Step E1 - Assess and compare the data extraction methods across reviews
For this step, Jadad states, “If reviews differ [in outcomes reported], the decision-maker should identify the review that takes into account the outcome measures most relevant to the problem that he or she is solving.” We interpret this step as selecting the SR that (A) matches the Discordant Review’s primary outcome.
Jadad then writes that SRs that conduct independent extractions by two reviewers are of the highest quality. We therefore decided that SRs that (B) used an independent data extraction process using two SR authors should be chosen. If a ROBIS assessment is done, then the latter point will be mapped to ROBIS 3.1. “Were efforts made to minimise error in data collection?”
Decision rules:
#1. Reviews that meet criteria A and B are highest in our hierarchy
#2. Reviews that meet criteria A are second highest in our hierarchy
#3. Reviews that meet criteria B are third highest in our hierarchy
Step E2 – Assess and compare clinical heterogeneity of the included RCTs across reviews
Clinical heterogeneity is assessed at the SR level by examining the clinical, public health, or policy question pertaining to the primary outcome and the eligibility criteria PICO elements of each included RCT to see if they are sufficiently similar. If the PICO across RCTs are similar, then clinical heterogeneity is minimal, and SRs can progress with pooling study results in a meta-analysis. If a ROBIS assessment is done, this question is mapped to ROBIS 4.3 “Was the synthesis appropriate given the nature and similarity in the clinical, public health, or policy questions, study designs, and outcomes across included SRs?”
If a SR states that (A) they assessed for clinical (e.g., PICO) heterogeneity across RCTs (in the methods or results sections), then this will be the SR that is chosen at this step. Example of a SR reporting a clinical heterogeneity assessment: "If we found 3 or more systematic SRs with similar study populations, treatment interventions, and outcome assessments, we conducted quantitative analyses (Gaynes 2014)". If authors reported and described clinical heterogeneity in the manuscript, then rule (B) authors that judged the clinical heterogeneity assessment to be minimal or low with rationale, will be chosen at this step.
Decision rule:
#1. Reviews that meet criteria A and B are highest in the hierarchy
#2. Reviews that meet criteria A are second highest in our hierarchy
Step E3 – Assess and compare data analysis methods across reviews
Jadad et al. are purposefully vague when describing how to judge whether a meta-analysis was appropriately conducted. T his step was interpreted as reviews that conducted: (A) an appropriate weighted technique to combine study results (i.e. fixed or random effects model) and (B) an investigation of statistical heterogeneity (i.e. by reporting I2, tau2, or chi2) (Figure 2).
Decision rules for if the presence or absence of heterogeneity is present in the meta-analysis:
#1. Reviews that meet criteria A and B are highest in our hierarchy
#2. Reviews that meet criteria A only are second highest in our hierarchy
#3. Reviews that meet criteria B only are third highest in our hierarchy (this decision can be ignored if heterogeneity is not observed)
Decision rules for Step E.
#1. Reviews that meet Step E1, E2, and E3 are highest in our hierarchy
#2. Reviews that meet Step E1 and E2 second highest in our hierarchy
#3. Reviews that meet Step E1 third highest in our hierarchy
#4. Reviews that meet Step E2 and E3 fourth highest in our hierarchy
#5. Reviews that meet Step E2 fifth highest in our hierarchy
#6. Reviews that meet Step E3 sixth highest in our hierarchy
Note: Reporting only Steps E1, E2 or E3 is not considered a systematic approach to evidence synthesis.
Figure 2. Our Approach to Operationalizing Step E of the Jadad Algorithm.
3.7.5 Step F - Select the review with the lowest risk of bias, or the highest quality
From the risk of bias/quality assessment conducted through Step D, we will choose the SR with the lowest risk of bias judgement, or highest quality assessment rating. ROBIS contains a last phase where reviewers are asked to summarise concerns identified in each domain and describe whether the conclusions were supported by the evidence. Based on these last decisions, a final SR rating will be made based on high, low or unclear risk of bias. For our Jadad assessment, we will choose a binary rating of either high risk or low risk of bias. Any SRs assessed as ‘Unclear’ risk of bias will be deemed as high risk. When using the assessments of risk of bias/quality of SRs from the included Discordant Reviews, we will choose the rating of the authors. If uncertainty exists, we will re-assess the included SRs using ROBIS.
3.7.6 Step G - Do the reviews have the same eligibility criteria?
If the SRs do not include the same trials, then decision-makers are directed to turn to Step G – assess whether the SRs have the same eligibility criteria (Figure 3). The Discordant Review may contain text in a methods section, or a characteristics of included SRs table where the PICO eligibility criteria can be extracted and assessed. If this is not the case, then the PICO eligibility criteria will be extracted from the included SRs by two authors independently and then compared to resolve any discrepancies.
Figure 3. Our Approach to Operationalising Step G of the Jadad Algorithm
2.8.7 Step H - Assess and compare the search strategies and the application of the eligibility criteria across reviews
If the SRs contain the same eligibility criteria, then Step H is to assess and compare the search strategies and the application of the eligibility criteria across SRs (Figure 4).
Step H1 - Assess and compare the search strategies across reviews
In this step, Jadad et al.’s recommendations are vague, although they make reference to comprehensive search strategies as being less biased. We interpret this step as authors explicitly describing their search strategy such that it can be replicated. To meet this interpretation, our criteria are that SRs: (A) search 2 or more databases, (B) search the grey literature; and (C) include a full search algorithm (may be attached as an appendix or included in the manuscript).
Decision rules:
#1. Reviews that meet criteria A, B and C are highest in our hierarchy
#2. Reviews that meet criteria A and B are second highest in our hierarchy
#3. Reviews that meet criteria A and C are third highest in our hierarchy
#4. Reviews that meet criteria B and C are fourth highest in our hierarchy (unlikely scenario)
#5. Reviews that meet criteria A only are fifth highest in our hierarchy
SCENARIOS for Step H1
• 3 SRs are identified for our Jadad assessment
Criteria to choose a systematic SR at Step H1: (A) 2 or more databases – (B) searched grey literature --(C) full search in appendix
Scenario 1
Review 1: A and B, but not C (decision rule #2)
Review 2: A and B but not C (decision rule #2)
Review 3: A and C, but not B (decision rule #3)
Conclusion: No SR meets ALL of our criteria; which do we choose? Based on our decision rules, we choose BOTH Review 1 and 2
Scenario 2
Review 1: A, but neither B nor C (decision rule #5)
Review 2: A and B, but not C (decision rule #2)
Review 3: Neither A, B, nor C (does not report the search methods)
Conclusion: No SR meets ALL of our criteria; which do we choose? Based on our decision rules, we choose Review 2
Step H2 - Assess and compare the application of the eligibility criteria across reviews
In this sub-step, Jadad indicates that we should choose the SR with the most explicit and reproducible inclusion criteria, which is ambiguous. Jadad states, “SRs with the same selection criteria may include different trials because of differences in the application of the criteria, which are due to random or systematic error. Decision-makers should regard as more rigorous those SRs with explicit, reproducible inclusion criteria. Such criteria are likely to reduce bias in the selection of studies” [22]. We did not know if this meant clearly reproducible PICO eligibility criteria, as this would be a repeat to Step G, whether the eligibility criteria were applied consistently by SRs (i.e. compare eligibility criteria to included RCTs’ PICO to see if they indeed met the eligibility criteria), or if this meant (A) independently screening of title, abstracts, and full text against the eligibility criteria by two reviewers. We selected the latter criteria when choosing from the included SRs in an Discordant Review.
Decision rules:
#1. Reviews that meet criteria A is highest in our hierarchy
Decision rules for Step H:
#1. Reviews that meet Step H1 and H2 highest in our hierarchy
#2. Reviews that meet Step H1 second highest in our hierarchy
#3. Reviews that meet Step H2 third highest in our hierarchy
Figure 4. Our Approach to Operationalising Step H of the Jadad Algorithm.
3.7.8 Step I – Assess and compare the publication status, quality, language restrictions of the included RCTs, and analysis of data on individual patients
If the SRs do not have the same eligibility criteria, then the next step, Step I, is to assess and compare the publication status, quality, language restrictions of the included RCTs, and analysis of data on individual patients across the SRs (Figure 5). This step maps to ROBIS item 1.5, namely, “Were any restrictions in eligibility criteria based on appropriate information sources (e.g. publication status or format, language, availability of data) [40]?”
Step I1 – Assess and compare the publication status of the included RCTs across reviews
In the absence of clear guidance, we interpret this step as “choose the SR that searches for and includes both published and unpublished data (grey literature).” Published studies are defined as any study or data published in a peer-reviewed medical journal. Unpublished data is defined as any information that is difficult to locate and obtained from non-peer-reviewed sources such as websites (e.g. World Health Organisation website, CADTH), clinical trial registries (e.g. clinicaltrials.gov), thesis and dissertation databases, and other unpublished data registries (e.g. LILIACS). Our interpretation is that SRs are chosen at this step that search for: (A) studies published in peer-reviewed medical journals, and (B) reports/documents/content that are not published in medical journals.
Decision rules:
#1. Reviews that meet criteria A and B are highest in our hierarchy
#2. Reviews that meet criteria A are second highest in our hierarchy
#3. Reviews that meet criteria B are third highest in our hierarchy
Note: Reporting only A or B is not considered a systematic search.
Step I2 – Assess and compare the methods used to assess the quality of the included RCTs across reviews
In this step, the Jadad paper recommends assessing the appropriateness of the methods used to assess the quality of the included RCTs across SRs. This item maps to ROBIS item 3.4, “Was the risk of bias/quality of RCTs formally assessed using appropriate criteria?” Here we interpret this item as to whether the SR authors used the Cochrane risk of bias tool (version 1 or 2). All other RCT quality assessment tools are inappropriate because they are out of date and omit important biases (e.g. Agency for Healthcare Research and Quality (AHRQ) 2012 [41] omits allocation concealment). However, the Cochrane risk of bias tool was only published in October 2008. Therefore, we applied a decision rule: for SRs dated 2012 (giving one year for awareness of the tool to reach researchers) and later, the Cochrane risk of bias tool is considered the gold standard. For SRs dated 2009 or earlier, we considered the Jadad [42] and Schulz [43] scales to be the most common scales used between 1995 and 2011. Other tools will be considered on a case-by-case basis.
As a decision hierarchy, to meet the minimum criteria for this step, a SR will have (A) assessed the risk of bias of RCTs using any tool or approaches, and (B) used the Cochrane risk of bias tool version 1 or 2 (if dated 2009 or later). If several SRs are included that meet these two criteria, the SR that (C) integrates the risk of bias assessments into the results or discussion section (i.e. discusses the risk of bias in terms of high and low risk of bias studies, reports a subgroup or sensitivity analysis) will be chosen.
Decision rules:
#1. Reviews that meet criteria A, B and C are highest in our hierarchy
#2. Reviews that meet criteria B and C are second highest in our hierarchy
#3. Reviews that meet criteria A and B are third highest in our hierarchy
#4. Reviews that meet criteria A and C are fourth highest in our hierarchy (unlikely scenario)
#5. Reviews that meet criteria A only are fifth highest in our hierarchy
SCENARIOS for Step I2
• 3 SRs are identified for our Jadad assessment
Scenario 1
Review 1: A and B but not C (decision rule #3)
Review 2: A and B but not C ( decision rule #3)
Review 3: A and C, but not B (decision rule #4)
Conclusion: No SR meets ALL of our criteria; which do we choose? Based on our decision rules, we choose BOTH Review 1 and 2
Scenario 2
Review 1: A, but neither B nor C (decision rule #5)
Review 2: A and B, but not C (decision rule #3)
Review 3: Neither A, B, nor C (does not report the search methods)
Conclusion: No SR meets ALL of our criteria; which do we choose? Based on our decision rules, we choose Review 2
Step 13 - Assess and compare any language restrictions across reviews
In this step, Jadad indicates that SRs with (A) no language restrictions in eligibility criteria should be prioritised and chosen over those that only include English language RCTs. This step maps to ROBIS item 1.5, namely, “Were any restrictions in eligibility criteria based on sources of info appropriate (e.g. publication status or format, language, availability of data)?”
Decision rule:
#1. Reviews that meet criteria A are highest in our hierarchy
Step I4 – Choose the analysis of data on individual patients
If (A) an individual patient data meta-analysis was identified in the Discordant Review, Jadad et al. recommend this SR be chosen over SRs with pairwise meta-analysis.
Decision rule:
#1. Reviews that meet criteria A are highest in our hierarchy
Decision rules for Step I:
#1. If there is an IDP meta-analysis (Step I4), then this SR is the highest in our hierarchy
#2. Reviews that meet Step I1, I2, and I3 are second highest in our hierarchy
#3. Reviews that meet Step I1 and I2 third highest in our hierarchy
#4. Reviews that meet Step I2 and I3 fourth highest in our hierarchy
#5. Reviews that meet Step I1 and I3 fifth highest in our hierarchy
#6. Reviews that meet Step I1 is sixth highest in our hierarchy
#7. Reviews that meet Step I2 is seventh highest in our hierarchy
#8. Reviews that meet Step I3 is eighth highest in our hierarchy
Note: Reporting only Steps I1, I2 or I3 is not considered a systematic approach to evidence synthesis.
Figure 5. Our Approach to Operationalising Step I of the Jadad Algorithm
3.8 Study outcomes
3.8.1 Evaluation of whether the Discordant Review authors:
- Used the Jadad decision tree to assess discordance
- Examine and record reasons for discordance (i.e. authors did not use Jadad)
- Use other approaches to deal with discordance (specify)
- Present discordance in tables and figures.
3.8.2 Results from our discordance assessment and Discordant Review authors’ assessments:
- Utility: Is the Jadad decision tree easy to use? (see section 2.9.3)
- Efficiency: How much time does it take to do one Jadad assessment?
- Comprehensiveness: is the Jadad algorithm comprehensive? Is it missing methods that might explain discordance (e.g. publication recency)?
- Jadad cohort: Frequency of disagreement or agreement across Jadad assessments between (a) Discordant Review authors’ assessment, and (b) our assessment (i.e. choosing the same SR).
- Non-Jadad cohort: Frequency of disagreement or agreement assessments between (a) Discordant Review authors’ assessment, and (b) our assessment (i.e. choosing the same SR).
- Comparison of Discordant Review authors stated sources of discordance and our identified sources of discordance.
3.8.3 “Ease of use” outcome measure
Each Jadad assessment will be assessed for “ease of use” by each assessor. Each Jadad assessment will be rated and colored (green, yellow, red) based on how easy or difficult the assessment was judged to be for the user. The rating is based on the following rubric:
- The step can be accomplished easily by the reviewer, due to low cognitive load or because it’s a recognised method or approach.
- The step requires a notable degree of cognitive load by the reviewer but can generally be accomplished with some effort.
- The step is difficult for the reviewer, due to significant cognitive load or confusion; some reviewers would likely fail or abandon the task at this point.
The lower the score, the easier the step is to complete.
3.9 Data extraction
3.9.1 Discordant review level extraction
The outcomes from section 3.8 will be extracted, along with the following information from the Discordant Review:
1. Study characteristics (lead author’s name, publication year).
3. Clinical, public health, or policy question (objectives, health condition of treatment, PICO eligibility criteria: participant, intervention/comparison, and primary outcome).
4. Methods (how the Discordant Review authors assessed discordance among the SRs [Jadad or other approach], how they operationalised Jadad, steps where they identified discordance, number of included SRs, type of SR (e.g. individual patient data meta-analysis, SR with narrative summary, SR with meta-analysis of RCTs), type of analysis (narrative summary or meta-analysis), risk of bias/quality assessment (e.g. AMSTAR, AMSTAR 2, or ROBIS tool), risk of bias judgments, and whether risk of bias/quality assessment was integrated into the synthesis.
5. Results (results of discordance assessment, effect size and confidence intervals (CIs), number of total participants in treatment and control groups, number of total events in treatment and control groups), direction of study results (favourable or not favourable).
6. Conclusions (difference between results and conclusions defined as if they disagreed in direction (results, not favourable; conclusions, favourable), authors’ result interpretation (quote from abstract and discussion section about the primary outcome result and conclusion).
3.9.3 SR level extraction
The outcomes from section 3.8 will be extracted, along with the following information from the included SRs:
1. Study characteristics (lead author’s name, publication year).
2. Clinical, public health, or policy question (objectives, health condition of treatment, PICO eligibility criteria: participant, intervention/comparison, and primary outcome; language restrictions and restrictions on publication status in eligibility criteria; citation of previous SRs/meta-analyses in background or discussion).
3. Search methodology (the name and number of databases searched, grey literature search details, the search period, language restrictions, restrictions on publication status, included full search in an appendix).
4. Methods (number and first author/year of included RCTs, effect metric (OR, RR, MD) and CIs, whether SR authors assessed the clinical (PICO) heterogeneity across RCTs (in the methods or results sections), analysis method (appropriate weighted technique to combine study results [i.e. used a fixed or random random-effects model], investigation of statistical heterogeneity [i.e. by reporting I2, tau2 or chi2], and if heterogeneity is present, then the authors investigated the causes of any heterogeneity [i.e. by reporting subgroup, sensitivity, or meta-regression analyses]), risk of bias/quality assessment (e.g. Cochrane risk of bias tool v1 or v2), risk of bias/quality judgment for each RCT, and whether the RCT quality/risk of bias assessment was integrated into the synthesis; 2 reviewers independently screened studies, extracted data and assessed risk of bias with process for resolving discrepancies found when comparing.
5. Results (effect size and CI, number of total participants in treatment and control groups, number of total events in treatment and control groups), direction of study results (favourable or not favourable).
6. Conclusions (difference between results and conclusions defined as if they disagreed in direction (results, not favourable; conclusions, favourable), authors’ result interpretation (quote from abstract and discussion section about the primary outcome result and conclusion).
Two authors will extract studies independently at full-text, and in the case of discrepant decisions, will discuss until consensus is reached.
3.10 Data analysis
We will assess and compare our outcomes (a) narratively for qualitative data, (b) using frequencies and percentages for categorical data, and (c) using median and interquartile range (IQR) for continuous data. Our analysis will be organised by our study outcomes in tables and in figures. We will discuss differences in the assessment of discordance across Discordant Reviews using Jadad and Discordant Reviews not using Jadad.