![[object Object]](https://images.prismic.io/researchsquare/53cbecf8-9778-466c-bb34-7e4b829d534a_Untitled+design+%282%29.png?auto=compress,format&rect=0,0,600,400&w=600&h=400)
Part I of this series provided an overview of the importance of critically appraising research to understand if the results can be trusted. Here, we describe a critical appraisal framework for evaluating bias in research.
Cochrane is a leading evidence-based medicine organization based in London, England, that has a global collaboration of researchers working to advance healthcare evidence and recommendations. Their assessment tools are considered benchmarks for the critical appraisal of bias in clinical research. Of the types of clinical research, randomized controlled trials are the gold standard. In these trials, participants are allocated into groups using a random process that helps balance confounding factors that may bias the results in favor of one group over the other. Cochrane’s instrument for evaluating randomized controlled trials (RoB 2) assesses bias using 22 questions across five domains, which are ultimately consolidated to derive an Overall Risk of Bias, as follows:
Domain 1: Risk of bias arising from the randomization process
Domain 2: Risk of bias due to deviations from the intended interventions (effect of assignment to intervention)
Domain 3: Missing outcome data
Domain 4: Risk of bias in measurement of the outcome
Domain 5: Risk of bias in selection of the reported result
To use RoB 2, a critical appraiser would answer a series of questions within each domain that determine a bias classification for that domain of low, some concerns, or high. Each of the domain-level classifications determine an overall risk of bias of low, some concerns, or high for the study as a whole. Low risk of bias is assigned when all four of the domains are classified as having a low risk of bias. Some concerns is assigned when at least one of the domains is classified as having some concerns, but none of them have been classified as having a high risk of bias. Finally, high risk of bias is assigned when at least one of the domains has a high risk of bias, or, multiple domains were classified as having some concerns.
If it is determined that a trial has some concerns or a high risk of bias, does that mean the outcomes from the trial cannot be trusted? No. The results could still be valid. However, the interpretation of clinical research is based around probabilities. If the trial’s methods are poor, there is a greater likelihood that the results do not estimate the true sample mean and/or that the results will not translate from the studied population to another population.
While the risk of systematic errors is only one component necessary to evaluate the validity of a study, along with design and measurement errors, systematic errors are considered among the most detrimental errors that affect a study’s outcomes. Hence, on its own, the overall risk of bias judgement is a proxy that indicates the extent of trust and faith that researchers, clinicians, and patients should invest in the results of the trial. For example, a study with a high risk of bias means that the results of the study have an increased chance of deviating away from the truth and trust should not be heavily placed in the results. In other words, there is a high risk that the estimated sample mean does not represent the true population mean.
The RoB 2.0 tool is complex. For individuals not versed in research methods, the following is a simplified checklist that can be used to examine if a clinical trial can be trusted. The checklist highlights six components that should be present in a clinical trial. If all of these components are present and well conducted, as described, faith can be more easily invested in the results:
1) The trial was randomized, using a true randomization method such as:
➜Random number table
➜Computer-based random number generator
➜Shuffling cards or envelopes
➜Tossing a coin, throwing dice, or drawing lots
2) The researchers concealed the allocation process when randomizing the participants, using a procedure such as:
➜Central allocation (including telephone, web-based and pharmacy-controlled randomization)
➜Sequentially numbered drug containers of identical appearance
➜Sequentially numbered, opaque, sealed envelopes
3) Participants and investigators were blind (masked) to knowledge of the intervention that was received.
➜Blinding reduces the risk that knowledge of the intervention, rather than the intervention itself, could affect the outcomes. Participants’ and/or investigators’ belief in an intervention may be powerful enough to bias the results. Hence, both participants and investigators should be blinded.
➜Blinding is not always possible, such as during a surgical trial where both the participants receiving the surgery and the investigators conducting the surgery know whether it has been performed.
4) Outcome assessors (researchers analyzing the results) were blind to the interventions the participants received.
➜This form of blinding may not be possible as some outcomes require knowledge of the intervention received. Thus, blinding of outcome assessors is not a hard-and-fast “must” for all clinical trials, and its necessity should be evaluated on an individual basis.
5) All outcome data should be reported.
➜If participants withdrew from the study, but their data is not analyzed, the result is attrition bias, defined as systematic differences between groups in withdrawals from a study.
➜Withdrawals commonly occur. Ideally, researchers will have used an “intention-to-treat” (ITT) analysis method to analyze all participants regardless of whether they withdrew from the study. ITT analyses are the best practice under any circumstance but are especially important when withdrawals occur.
6) Selective outcome reporting should not have occurred.
➜Systematic differences between reported and unreported findings is referred to as selective outcome reporting, also known as reporting bias. Ideally, the researchers will have registered a study protocol that prespecifies the outcomes they intended to study and analyze before beginning the trial. By comparing the protocol to the published article for differences between outcomes, reporting bias can be detected. If a study protocol is not available but it is clear that the published reports include all expected outcomes, reporting bias may not be present, however, convincing evidence of this nature may be rare.
While critical appraisal of a study’s risk of bias is the foremost process necessary to evaluate the validity of clinical research, additional methodological components are necessary to consider; these will be detailed in Part III of this series.