- A systematic review (SR) is research that collates all empirical evidence around a topic to answer a question and evaluate the collective results, bias, strength, and consistency of the evidence.
- A meta-analysis (MA) uses statistical techniques to quantitatively combine results from studies included in an SR to assess the statistical significance, direction (in favor or against the tested intervention, risk factor, or exposure), and heterogeneity (the variation in outcomes between studies) of the results when viewed as a collective group. MAs are an alternative to narrative-based discussion of research results.
- An SR must be performed in order to conduct an MA. However, an MA does not have to be performed with an SR, and sometimes it is not possible to conduct an MA due to lack of appropriate data to statistically combine results.
- Well conducted SR/MAs are considered the most reliable form of evidence, at the top of the evidence hierarchy, because they draw conclusions from the totality of the literature around a topic . They are the fundamental research underlying evidence-based medicine.
- The agreement, or disagreement, between studies is demonstrated in SR/MAs, allowing a complete and unbiased perspective on a topic.
- The bias present in individual studies included in SR/MAs is evaluated, which helps understand if the results and conclusions from each trial can be trusted, both individually and when combined.
- Explicit methods for conducting the SR/MA are prespecified, including defining the eligibility criteria of the studies that will be included in the SR and the outcomes that will be collected and analyzed. Defining these components before conducting the study minimizes bias that may occur when conducting the systematic review that helps ensure that the totality of the literature is discovered and analyzed appropriately so that correct conclusions are drawn.
- SR/MAs are less vulnerable to biases that are present in another commonly performed review, the narrative literature review (NLR). Systematic methods are not used when performing NLRs. In fact, researchers may “cherry-pick” studies and results to present in NLRs by excluding eligible studies that should have been included, leaving the results incomplete and the conclusions biased.
- By statistically combining the results from all trials in an MA, more precise estimates of the effects of interventions can be understood and compared to those derived from individual studies alone. In turn, MAs optimize clinical recommendations and practice guidelines.
- Although SR/MAs are considered the highest level of evidence and systematic methods are used in an attempt to minimize bias, SR/MAs can also be biased if not properly conducted.
- Many SR/MAs arrive at the conclusion that there is no evidence an intervention is more effective than comparators. An uncertain conclusion does not necessarily mean that the intervention does not work. Rather, it could mean that the studies had severe methodological weaknesses and that given the current state of the literature, the researchers do not have confidence in the effects of the intervention until additional high-quality research is performed.
- Even if SR/MAs have an uncertain conclusion, the intervention may still be implemented in clinical practice. It is possible the intervention works in some populations or that an expert panel arrives at a consensus conclusion that differs from the results in a SR/MA. In such cases, these expert recommendations are used to inform clinical guidelines instead of the SR/MA, which - it can be argued - is an inappropriate and non-evidence-based medicine approach. Additionally, clinicians may judge that the benefits of using the intervention outweigh the risks or uncertainty in the effects demonstrated in an SR/MA. Consequently, the intervention may be promoted and used in clinical practice despite lack of evidence from an SR/MA, which again, may be inappropriate.
What to watch for
The following broadly outlines differences between the systematic reviews and narrative literature reviews:
- Based on empirical evidence.
- Objective and predefined inclusion and exclusion criteria.
- Reproducible methods.
- Describes a definitive search strategy that can be reproduced.
- A formal and objective method is used to assess the risk of bias and validity of the studies.
- The results are combined in a prespecified and formal way.
Narrative Literature Reviews
- Based on opinion and/or “cherry-picked” evidence.
- Inclusion and exclusion criteria not defined.
- Irreproducible methods.
- No definitive search strategy or way to determine how the researchers identified the studies for inclusion.
- The risk of bias and validity of the study was not objectively assessed. The researchers’ subjective opinions constitute the analysis.
- The results are not strategically combined but rather linked together as a “story.”
How the Dissemination Crisis Led to the Replication Crisis
Explainer: What is a Cell Line?
In His Own Words: a Q&A on Preprints with Medical Scientist Pritam Sukul
Research Square Methods Badge: A Case Study