This section has two parts. The first part identifies key factors and considerations that can inform the decision to select the most appropriate review type to answer a research question about intervention effects in an HTA. The second part organizes those key factors and considerations into a summary and a flowchart that can guide the decision-making process.
Part 1. Key Factors and Considerations
In total, nine key factors and six considerations that are important for selecting a specific review type to conduct for evaluating intervention effects were identified through reflections on experiences with clinical reviews in past HTA projects at CADTH and IHE. The literature review found these factors and considerations to be in good agreement with existing guidance from academia, SR organizations, and other HTA agencies. In total, 19 papers were identified as relevant through the literature review. They were from Joanna Briggs Institute,27 the Cochrane Collaboration,2,5 the Agency for Healthcare Research and Quality,15,18,19,28 the Belgian Health Care Knowledge Centre,29 the World Health Organization,30 eight different HTA agencies,31 the Centre for Reviews and Dissemination at the University of York,8 and other research groups4,7,10,17,32-35 (see Appendix 3). The feedback received from other HTA researchers across Canada also found these factors and considerations to be representative of the experiences of different organizations and applicable across different decision-making contexts.
The nine key factors and six considerations each fell into one of two categories: the evidentiary needs of the planned review and the state of the existing literature. The key factors and considerations are presented in Table 1 and described below, with supporting citations. Of note, not all three review types are discussed or discussed to the same extent under each factor, as not all factors and considerations are relevant or deterministic in favouring or ruling out each review type. Also, many of the key factors and considerations are interdependent, as the evidentiary needs of a planned review must be balanced with the state of the existing literature. Therefore, many of the factors and considerations are not sufficient on their own to help select one review type over another, which motivated us to organize them into a summary and a corresponding flowchart in the following section. Further, we kept in mind the unique challenges that overviews present, stemming from having to rely on SRs, instead of primary studies, as the unit of analysis, and therefore possibly requiring more resources, compared to SRs,15,19 and tried to balance them with the potential benefits obtained (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).
Reviewers who plan to use these key factors and considerations to help select the most appropriate review type for their HTA will need an understanding of both the evidentiary needs of the planned review and the state of the existing literature. Specifically, there must be a clear research question with a well-defined scope. Further, a scoping exercise should be undertaken to gain a general idea of the quantity, quality, and other characteristics of relevant SRs and primary studies that are available. While it may be difficult to obtain a comprehensive understanding of the literature through the scoping exercise, and reviewers will likely work with incomplete information, the guidance provided in this paper is intended to help reviewers think through, and plan for, the issues that they are likely to encounter when deciding which review type to conduct. To that end, reviewers may find it difficult to answer the questions provided below with certainty (e.g., instead of “yes” or “no”, “likely yes”, “likely no”, or even “unclear”), especially when the existing literature is vast or complex. Nevertheless, the scoping exercise could be viewed as an “upfront” investment in the conduct of an HTA that need not be wasted (e.g., information learned through the scoping exercise can be “reused” during the review process) and may end up saving resources (e.g., by identifying an existing SR and eliminating the need for a new SR) or achieving other meaningful goals (e.g., summarizing the existing literature at higher levels or avoiding or minimizing duplication or redundancy in research).
I. Evidentiary needs of the planned review
2. Review scope: Is the scope of the research question broad?
A de novo SR and an update of an SR are identical or similar in scope,4 which tends to be focused and narrow.7,34 An overview of SRs, on the other hand, tends to have a broader scope.7,8,10,32,33,35 When expressed in terms of the Population, Intervention, Comparator, and Outcome (PICO) framework, a de novo SR or an update of an SR tends to have narrowly defined PICO elements, while an overview tends to have expanded upon one of the PICO elements. As described by the Cochrane Collaboration, overviews are appropriate for addressing research questions relating to:
- The same intervention used for different conditions or populations (i.e., broader P);
- Different interventions used for the same condition or population (i.e., broader I);
- Different approaches to the application of the same intervention used for the same condition or population (i.e., broader I);
- The same intervention used for the same condition or population but for different outcomes or time points (i.e., broader O); or
- Adverse effects of an intervention for one or more conditions or populations (i.e., broader O or P).27,32,34
In other words, for an overview, there are two or more relevant SRs available that each covers one “part” of a broad research question, with all SRs combined resulting in perfect or almost perfect coverage.5,7,10,27,32-35
Consideration I: If the planned review has narrowly defined PICO elements, a de novo SR or an update of an SR may be the most appropriate review type. If the planned review has a broader scope that expands upon one of the PICO elements, an overview of SRs may be the most appropriate review type.
2. Review objective and analytic approach: Is quantitative combination of findings needed to provide a summary measure or to rank interventions?
A de novo SR, an update of an SR, and an overview of SRs all intend to answer a specific research question,1 using explicit, systematic methods.34 However, the unit of searching, inclusion, data extraction, analysis, and synthesis varies from the primary study – for a de novo SR and an update – to the SR – for an overview.
A de novo SR or an update of an SR – having access to primary study-level data – is best suited for quantitative combination of findings through meta-analyses (MAs), indirect treatment comparisons (ITCs), or network meta-analyses (NMAs) to provide a summary measure or answer a question about which intervention works best or is the safest.
With the SR as the unit of analysis, there are limitations in the analytic capabilities of an overview, and hence the research objectives it can fulfill. In an overview, quantitative combination of findings is more difficult, compared to narrative incorporation of findings, without going back to the primary study reports.15,27 This is especially true if relevant SRs are of low quality in methodology or reporting (leading to errors or missing data) or if they overlap significantly in their included primary studies (giving too much statistical power to certain studies).15,27 Further, unless reviewers have thoroughly examined the transitivity assumption (i.e., that the studies making different direct comparisons are sufficiently similar in all respects other than the treatments being compared) and found it to be valid, which is very difficult to do using SR reports, ITCs or NMAs should not be conducted in overviews.5,15,31,34 Therefore, an overview of SRs – relying on review-level data – is best suited for examining a body of SR evidence to provide overall trends in research findings or answer a question about which interventions are effective or safe or for exploring if and why the evidence base on a topic or question is heterogeneous.27 ITCs and NMAs in overviews are explicitly discouraged, and informal indirect comparisons should also be avoided.5
Consideration II: If the planned review requires quantitative combination of findings through MAs, ITCs, or NMAs to provide a summary measure or to rank interventions, a de novo SR or an update of an SR should be considered instead of an overview of SRs. In an overview, a narrative, not quantitative, synthesis should be conducted.
II. State of the existing literature
3. Relevance: Are there one or more relevant SRs available?
To answer this question, database searches with appropriate SR filters or targeted searches for potentially relevant SRs may be needed. Potential SRs of interest could then be assessed on whether they meet eligibility criteria of the planned review, as specified by the PICO elements.28,29 If the scope of an SR is identical to or narrower than that of the planned review, the entire SR would be relevant. If its scope is broader, the subset of its findings that meets the eligibility criteria of the planned review would be relevant, if reviewers are able to extract the relevant results separately (e.g., from a subgroup analysis).
If there is no relevant SR, conducting a de novo SR is likely necessary.
For an update, there must be one or more relevant SRs available that each perfectly or almost perfectly covers the research question. It may be possible to integrate multiple SRs in an update,15,19 but this is discouraged due to logistical and methodological complexities, as it would likely require merging existing SRs first with each other and secondly with newly identified primary studies.1,5 Therefore, our recommendation is to handle multiple relevant SRs through either an overview of two or more SRs or an update of a single SR that is “best” in terms of relevance, quality, comprehensiveness, and currency.18
For an overview, there should be two or more relevant SRs available that each cover at least one “part” of a research question, with all SRs combined resulting in perfect or almost perfect coverage. There is no accepted minimum or maximum number of SRs required for an overview, although recommendations range from including a minimum of “two or more” SRs5 to “5-10 or more” SRs.29
Consideration III: The quantity of relevant SRs available may rule out certain review types for the planned review. If there is no relevant SR, no overview or update could be conducted. If there is only one relevant SR, no overview should be conducted.
4. Methodological quality: Are the relevant SRs of sufficiently high quality in methodology?
While it may sometimes be appropriate to include all relevant SRs in the planned review for completeness, it is often advised to assess the methodological quality of existing SRs and only use high-quality SRs in updates or overviews to build on high-quality literature.2,5,8,15,19,28,31 In the absence of high-quality SRs, replication in the form of a de novo SR may be deemed appropriate to produce a higher-quality SR.2,17,36
To assess methodological quality, relevant SRs should be critically appraised for their methodological rigour.17,29 Established critical appraisal tools, such as AMSTAR 237 or ROBIS,38 can be used.17,29 There is no broadly accepted threshold for what is considered sufficiently high quality in methodology for a relevant SR to be used in a planned review, although there is guidance for what is good, fair, or poor quality.8,18 The threshold for sufficiently high quality in methodology may vary depending on the purpose of the review, how the relevant SRs will be used, whether there are high-quality SRs available, and what resources are available for the planned review. Nevertheless, criteria for determining whether relevant SRs are of sufficiently high quality in methodology should be established a priori for each planned review to minimize bias.28
5. Reporting quality: Are the relevant SRs of sufficiently high quality in reporting?
Poor reporting in existing SRs would make it difficult not only to extract data but also to decide whether the SRs are relevant, of high methodological quality, comprehensive, and up to date, and hence, useful for the planned review. Therefore, high quality in reporting is desirable,15,19 especially if the SRs are to be used in overviews without going back to primary study reports.
To assess quality in reporting, reporting guidelines may be used,32 supplemented with additional criteria required for the planned review (e.g., exact data needed, such as point estimates and associated variability in a specific format). Available reporting guidelines include PRISMA Statement (for all SRs),39 SWiM (for SRs with narrative syntheses),40 and MOOSE (for SRs with MAs of observational studies in epidemiology).41 There is no broadly accepted threshold for what is considered sufficiently high quality in reporting for an existing SR to be used. However, at the least, it should be well written, describe the PICO elements and methods clearly, and report all necessary data in an easily extractable form.
Consideration IV: Ideally, only SRs of high quality in both methodology and reporting should be used for updates and, in many cases, for overviews. In the absence of high-quality SRs, replication in the form of a de novo SR may be appropriate.
6. Comprehensiveness: Are the relevant SRs comprehensive?
To answer this question, relevant SRs should be examined on whether they include all relevant PICO elements of the planned review either individually (i.e., a single SR is comprehensive on its own) or together (i.e., two or more SRs are comprehensive when combined). If they do not, using them in an overview may result in research gaps in the final review, as it is not recommended that an overview of SRs be supplemented with primary studies due to logistical and methodological complexities.1,5 Therefore, a research question that cannot be fully addressed with existing SRs due to important research gaps in those SRs may be best addressed by a de novo SR or an update of an SR (e.g., with a search for additional data from an expansion of one or more of the PICO elements or a re-analysis of the available data for a different outcome) if primary studies are available to fill those gaps. On the other hand, if two or more relevant SRs are available that when combined result in perfect or almost perfect coverage,5,7,10,27,32-35 an overview of SRs may be appropriate.
7. Currency: Are the relevant SRs up to date?
In general, an SR may be considered to be up to date when 1) there is no new evidence or 2) there is new information, but it is unlikely to change the review conclusions.34 However, the definition of currency varies from one field to another, as the acceptable time period from the last date of search will depend on various factors, such as the activity or advances in the field or in methods, the importance and urgency of the research question being addressed, and the level of uncertainty around the evidence base.4,8,19,42 Criteria for determining the currency or outdatedness of relevant SRs need to be established a priori for each planned review to minimize bias.28,43
For those overviews that have the objective to answer a research question about intervention effects (as opposed to the objective to describe a body of literature), all relevant SRs should be sufficiently up to date.5 Whether an SR is sufficiently up to date should be determined with the help of experts in the field of research who have the knowledge to speak to the various factors that define currency mentioned above. Therefore, a research question that cannot be fully addressed with existing SRs because they are out of date may be best addressed by an update of an SR – if there is a high-quality SR available – or by a de novo SR – if there is no high-quality SR available – that is, if new evidence is available to bring it up to date.8
8. New evidence: Are there additional relevant primary studies missing from the relevant SRs?
Additional primary studies that are relevant to the research question but not captured by relevant SRs may be identified during scoping or through expert input. They may be 1) newly published studies or 2) studies not identified by relevant SRs because the SRs had narrower scopes, searched fewer databases, or had limited information sources (e.g., no grey literature). Since an overview of SRs should not be supplemented with primary studies,1,5 a research question that cannot be fully addressed with existing SRs because additional primary studies would be needed for comprehensiveness or currency may be best addressed by an update of an SR – if there is a high-quality SR available – or by a de novo SR – if there is no high-quality SR available.
Consideration V: If relevant SRs are not comprehensive or are outdated and need to be supplemented with additional primary studies that are available, an overview of SRs may not be appropriate. Instead, an update of an SR – if there is a high-quality SR available – or a de novo SR – if there is no high-quality SR available – is recommended for consistency in the analytic approach at the study level.
9. Discordance in results: Are the findings of the relevant SRs discordant for unknown reasons?
If there are two or more relevant SRs addressing the same research question with matching PICO elements, and if they are of sufficiently high quality in both methodology and reporting, their results could then be compared. Results are concordant if they match with respect to the direction, magnitude, and statistical significance of the estimated intervention effects for the outcome(s) of interest and in terms of the interpretations and inferences made by the SR authors.44 Conversely, results are discordant if they differ in any of those aspects.44 Such differences may or may not be important enough to lead to different health care decisions.44
If the results of relevant SRs with matching PICO elements are discordant and if the differences matter for the planned review, reasons for discordance should be explored.15,18,19,30,35,44 Potential reasons for discordance include differences in the methods or timing of different SRs that lead to different sets of studies being included, data being extracted, analyses being conducted, or syntheses or interpretations being made.44 If reasons for discordant findings are clear, and if there is certainty around which discrepant results to trust15,30,35 – likely those of high quality in both methodology and reporting and most up to date – reviewers may choose one or more SRs that are most appropriate for the planned review and conduct an update or an overview. If reasons for discordant findings are unclear and if there is uncertainty around which discrepant results to trust,15,30,35 a de novo SR may be indicated, instead of using one or more of the conflicting SRs.15,19
Consideration IV: If the results of two or more relevant SRs with matching PICO elements are discordant, and reasons for discordance cannot be reliably determined, a de novo SR may be needed.
Part 2. Making the Decision
A flowchart is presented in Figure 1 and described below. This flowchart depicts the nine key factors and six considerations outlined above in the order that we see as helpful for selecting a specific review type to conduct for evaluating intervention effects in an HTA. The flowchart should be followed from top to bottom, first with Part A and then with Part B. Each question is indicated with a “Q” followed by a number that corresponds to the numbering of the key factors identified above. Both Part A and Part B begin with Q1 and Q2 (in blue boxes), which address the evidentiary needs of the planned review, and then proceed to Q3 through Q9 (in orange boxes), which address the state of the existing literature. Some of the questions from Q3 to Q9 are combined or modified for further investigation. Depending on the responses to the questions (“yes” or “no” in black circles), some review types are eliminated through this decision-making process (in yellow boxes), leading to one decision at the end (in green boxes).
If there is only a single relevant SR that is high quality in both methodology and reporting, comprehensive, and up to date, there is likely no need for a new SR of any kind for the specific research question (see D5 in Figure 1 in both Part A and Part B). In this case, the existing SR can and should be used instead of conducting a new SR. Similarly, if there are multiple such SRs without discordant results for unknown reasons, no new SR should be conducted, and the best existing SR can be used (see D7 in Figure 1 in both Part A and Part B). In all other cases, a decision on which review type to conduct must be made.
For some situations, there are considerations regarding the key factors identified above that are absolutely deterministic in ruling out certain review types, including the following:
- If the planned review requires quantitative combination of findings through MAs, ITCs, or NMAs, an overview should not be conducted (Consideration II; see D2 in Part A of Figure 1).
- If there is no relevant SR, an update or an overview could not be conducted (Consideration III; see D3 in Part A and D9 in Part B of Figure 1).
- If there is only one relevant SR, an overview should not be conducted (Consideration III; see D10 in Part B of Figure 1).
For most situations, however, it is the interplay of multiple considerations regarding the key factors that informs the decision on which review type to conduct. In these cases, while no review types can be completely ruled out, one review type may be favoured over the others, as follows:
- If the planned review has a narrow scope (Consideration I), instead of an overview (see D1 in Part A of Figure 1), it may be more appropriate to conduct a de novo SR (see D4 in Part A of Figure 1) or an update of an SR (see D6 and D8 in Part A of Figure 1), depending on whether one or more relevant SRs of high quality are available (Consideration IV) without discordant results for unknown reasons (Consideration VI). For example, if there is a single “best” relevant SR that is high quality in both methodology and reporting and comprehensive but needs an update with newly available data, new methods, or new analyses, with concordant results with other relevant SRs, an update may be the most appropriate review type. If there is no relevant SR at all or no relevant SR that is high quality in both methodology and reporting and comprehensive, a de novo SR may be needed.
- Conversely, if the planned review has a broader scope (Consideration I), and there are two or more relevant SRs available (Consideration III), an overview may be the most appropriate review type, especially if the relevant SRs are high quality in both methodology and reporting, comprehensive when combined, up to date, and concordant in their findings, with no need to be supplemented with additional primary studies (Considerations IV, V, and VI; see D12 in Part B of Figure 1). If one or more of these conditions on the existing literature are not met, reviewers should carefully consider whether an overview of SRs is the most appropriate review type. It may be more appropriate to conduct a de novo SR, especially if the number of relevant primary studies is manageable with given resources (see D4 and D13 in Part B of Figure 1), or an update if there is a single “best” relevant SR that is high quality in both methodology and reporting and comprehensive on its own (see D 6 and D8 in Part B of Figure 1).
The flowchart in Figure 1 outlines a decision-making process for selecting a review type to conduct, capturing all of the considerations from above.