We will follow the methodology proposed by Whiting [29], Sanderson [37] and Page [7] for creating systematically developed lists of quality items.
3.1 Eligibility criteria
There will be two types of studies included. Study type 1 are articles that present and describe items related to bias, reporting, or methodological quality of reviews with NMA. Items related to reporting will be retained because they can potentially be translated into a risk of bias item. For example, in the PRISMA-P guideline [38], one item asks whether study PICO [Population, Interventions, Comparisons, Outcomes] characteristics were used as criteria for determining study eligibility. Reporting of all outcomes in a protocol may prevent authors from only selecting outcomes that are statistically significant when publishing their systematic review. This PRISMA-P reporting item can then be translated into a bias item related to the “selective reporting” of outcomes [39]. Study type 2 are studies that assess the methodological quality in a sample of reviews with NMA.
Study Type 1 will meet any of these inclusion criterion:
- Articles describing items related to bias or methodological quality in reviews with NMA (e.g. Dias 2018 [40]); tools that only assess general aspects of systematic reviews without focusing specifically on NMA will be excluded (e.g. AMSTAR [21], AMSTAR 2 [24] or ROBIS [17]).
- Articles describing editorial standards for reviews with NMA (e.g. similar to the Cochrane MeCIR (Methodological standards for the conduct of new Cochrane Intervention Reviews) standards for systematic reviews [30]).
- Articles describing items related to reporting quality in reviews with NMA (e.g. PRISMA-NMA [25]).
- Articles identifying or addressing sources of bias and variation in NMA and published after PRISMA-NMA in 2014.
Study Type 2 will meet any of these inclusion criterion:
- Articles assessing the methodological quality (or risk of bias) of reviews with NMA (i.e. a sample of NMAs are assessed for methodological quality; e.g. Chambers 2015 [41]) using criteria that focus specifically on aspects of NMA not just on general aspects of systematic reviews.
We will include articles with any publication status and in any language, and where the co-authors are not fluent in the language, Google Translate will be used.
If through our main search, we identify a systematic review encompassing the eligible articles, or one aspect of the eligible article, we will use the results of the systematic review and only include primary studies published subsequent to the systematic review. For example, a review by Laws et al. in 2019 [5] identified all guidance documents for conducting an NMA from countries throughout the world. We therefore would not search for guidance documents published before the last search date of this review.
3.2 Search strategy
We will search Ovid MEDLINE (January 1946 to June 2020), the Cochrane library as well as the following grey literature databases: the EQUATOR Network (http://www.equator-network.org/reportingguidelines/), Dissertation Abstracts, websites of evidence synthesis organisations (Campbell Collaboration Cochrane Multiple Treatments Methods Group, CADTH, NICE-DSU, Health Technology Assessment International (HTAi), Pharmaceutical Benefits Advisory Committee, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, European Network for Health Technology Assessment, Guidelines International Network, ISPOR, International Network of Agencies for Health Technology Assessment, and JBI) as well as methods collections (i.e. Cochrane Methodology Register, AHRQ Effective Health Care Program). We will validate the MEDLINE strategy by using the PubMed IDs of ten included studies (identified by experts prior to our eligibility screening) and evaluating whether the strategy identified the PMIDs (Appendix 1).
A systematic search strategy will be developed by two methodologists (CL, PW) without limitations to publication type, status, language, or date to identify existing tools or articles. An information specialist will check the search strategy for MEDLINE Ovid and assess it using the PRESS (Peer Review Electronic Search Strategies) guidance [42]. The full search strategies for all databases and websites can be found in Appendix 1. To identify other potentially relevant studies, we will examine the reference lists of included studies. We will ask experts in methods for NMA to identify articles missed by our search. We will contact authors of abstracts to retrieve the full report or poster.
We will search the reference section of a bibliometric study of reviews with NMAs [43] and extract the name of the journals that publish NMAs. We will then contact their editors in chief and ask if they have any in-house editorial standards for reviews with NMA.
3.3 Process for screening, data extraction and analysis
The eligibility criteria will be piloted in Microsoft Excel by two reviewers independently on a sample of 25 citations retrieved from the search to ensure consistent application. After high agreement (>70%) is achieved, the Covidence [44] web-based tool (https://www.covidence.org) will be used by two reviewers to independently screen the citations based on the eligibility criteria. Disagreements will be discussed until consensus is reached. A third reviewer (CL) will arbitrate if disagreements cannot be resolved.
The data extraction form will be piloted by reviewers independently on a sample of five included papers to ensure consistent coding. Two independent authors will extract data on the characteristics of the studies, and items. Any disagreements will be arbitrated by a third author.
3.4 Data extraction
The sources will first be categorised by the type of article coded as per our inclusion criteria. A table of tool characteristics will be developed with the following headings: first author, year; type of tool (tool, scale, checklist, or domain-based tool); whether the tool is designed specific topic areas (specify); number of items; domains within the tool; whether the item relates to reporting or methodological quality (or other concepts such as precision, acceptability); how items and domains within the tool are rated; methods used to develop the tool (e.g. review of items, Delphi study, expert consensus meeting); and the availability of an “explanation and elaboration” [7].
Data will be extracted on items that are potentially relevant to the risk of bias or quality of reviews with NMAs. Items will be initially extracted verbatim.
3.5 Data analysis
The following steps will be used when analysing items:
- Map to ROBIS domains
Items will be mapped to ROBIS domains (study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings) and specific items within the domains. The rationale for mapping items to ROBIS is that it is the only tool to assess risk of bias in reviews. Items that do not clearly map to the existing ROBIS domains will be listed separately and grouped by similar concept. New domains may be created if items do not fit well into the established ROBIS domains.
- Split items so that each item only covers a single concept
Two or more concepts grouped in one item will be split so that each item covers a single concept. A rationale as to why the item was split will be described. For example, PRISMA-NMA item 15 (“Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)”) will be split into two items because this item is represented by two items in ROBIS in the synthesis and findings domain, namely “4.5 Were the findings robust, e.g. as demonstrated through funnel plot or sensitivity analyses?” and “4.6 Were biases in primary studies minimal or addressed in the synthesis?”.
- Group similar items
Items that are conceptually similar will be grouped together and noted with the source. We will classify items as relating to bias or other aspect of quality (e.g. reporting). When relevant, items related to reporting will be reworded into items related to bias in NMA review conclusions.
- Omit duplicate items (but keep these in a column in the table for transparency)
If items are worded vaguely or are unexplained, we will use an iterative process to interpret the item and ensure there is a mutual understanding of the item between authors when coding. The process will be iterative, and if any gaps in items related to bias in reviews of NMA are identified, a new item will be inferred.
The final list of items deemed unique will be retained. We will reword items as signalling questions, where an answer of “yes” suggests absence of bias. We will provide examples to illustrate the items and write a rationale and description of each item.
We will count the number of sources and unique items included. We will summarise the characteristics of included tools in tables and figures. We will calculate the median and interquartile range (IQR) of items across all tools and tabulate the frequency of different biases identified in the tools.
3.6 Patient and Public Involvement
Patients or the public were not involved in the design of our research protocol.