Science is glamorized as a perfectly conducted system under rigid standards and produces trusted results that withstand scrutiny. Unfortunately, this is not the case.
There are countless examples of where science fails. Most alarmingly, the failures come from clinical research where lives are at stake. For example:
1. Discrepancies in the literature are evident when comparing randomized controlled trials to observational studies on the same topic. Studies examining the antioxidant properties of vitamins have demonstrated inverse associations with all-cause mortality, cancer, and cardiovascular disease in observational studies. However, the protective nature of antioxidant supplementation disappeared when multiple randomized controlled trials were conducted that tested the same associations.
2. Inconsistencies occur between studies utilizing similar trial designs. Re-analysis of 37 randomized controlled trials demonstrated that 62% of results were changed after re-examining the results, including re-interpretation of the patients that should be treated and changes in the direction, magnitude of treatment effect, or statistical significance. Similarly, out of 49 highly cited original clinical trials, subsequent studies contradicted 16% of the original research, while 16% demonstrated weaker treatment effects.
3. Discrepancies may be due to errors in the conduct, reporting, and analysis methods. Re-analysis of 250 controlled trials demonstrated that treatment effects were overestimated when trials utilized inadequate concealment methods or failed to adequately report the concealment methods (P < 0.001).
The reproduction and replication crisis
These issues underlie the reproduction and replication crisis. According to the reproduction and replication crisis, between 35-90% of research investigating major domains such as psychology, genetics, oncology, and cardiovascular disease, and cancer, cannot be reproduced or replicated.
Aside from the immediate impact of erroneous results that render the research deceptive and untrustworthy, there are larger clinical implications that affect patients and care providers. Researchers, healthcare professionals, and the public are prevented from obtaining reliable evidence of the efficacy and effectiveness of interventions that negatively impacts their ability to make appropriate healthcare decisions.
Poor science can only be rectified by better science. While criticizing studies post-hoc through peer review and open access commenting helps identify issues within studies, this criticism cannot correct bad science that has already been performed. Better studies are required, and these trials need to be properly conducted and free from bias to ensure that the results are the closest estimation to the truth as possible.
Randomized controlled trials (RCTs) are the gold standard clinical trials. They attempt to establish causality between an intervention and outcome. Ideally, RCTs should be free from confounders (link to post) and bias. However, an RCT is not an RCT. In other words, there are good RCTs that have been properly conducted, and there are bad RCTs with poor methods and bias.
Below are six steps investigators need to perform to design and conduct an unbiased RCT:
1. Protocols
Write a protocol that prespecifies the methods that will be used and outcomes that will be measured. Protocols help reduce the risk of bias by holding investigators accountable to a predetermined plan. To deduce bias between the plan and the final research report, protocols can be compared to the final report to ensure that the methods and outcomes were followed as prespecified. Protocols also help protect against selective outcome reporting, which is choosing only the positive, statistically significant, and novel findings to report, that may lead to a biased perspective of the true outcomes if some are withheld from dissemination that may distort clinical conclusions.
2. Adequate number of participants
An adequate number of participants (the sample size) needs to be enrolled to detect a true difference between groups that provides the statistical confidence that the results are valid. If an inadequate number of participants are enrolled, the results may be incorrectly estimated. This leads to biased conclusions.
Depending on the research question and outcomes, a specified sample size is required based on the minimum clinically important difference, which is the clinical threshold that the results must meet to have the clinical importance that would change a patient's management. This means that if the intervention works, the participants will observe an effect that is valuable for their condition.
Many clinical trials have inadequate sample sizes. If significant effects are demonstrated, the results may be exaggerated and overestimated. If non-significant effects are demonstrated, and absence of evidence is demonstrated. In either case, the results cannot be trusted. Hence, it is imperative that:
- Adequate sample sizes are calculated prior to enrolling participants
- The specified number of participants are enrolled and randomized into groups
- All data from these participants are included in the analysis
3. Proper randomization
Participants need to be randomized into groups to ensure that all confounders between groups are equal and cannot bias the results in favor of one group over the other. The primary benefit of RCTs is that if properly conducted they can establish causality, that an intervention causes an outcome. Without proper randomization that equally balances confounders, causality cannot be established.
Appropriate randomization procedures include:
- Using a computer random number generator
- Using a random number table
- Tossing a coin
- Shuffling cards or envelopes
- Throwing dice
- Drawing of lots (straws, pebbles, etc.)
Inadequate methods of randomization include:
- Block randomization
- Odd or even date of birth
- An arbitrary rule, such as date of admission, hospital, or clinic record number
- Judgment of the clinician
- Preference of the participant
- Allocation based on laboratory tests
- Allocation by availability of the intervention
4. Conceal allocation sequence
The allocation sequence used to randomize participants needs to be concealed from the investigators and participants. This means that when participants are randomly allocated into each group, the procedure is concealed so that neither the investigators nor the participants know the group that participants are placed into. Appropriate concealment methods include:
- Central allocation (including telephone, web-based and pharmacy-controlled randomization).
- Sequentially numbered drug containers of identical appearance.
- Sequentially numbered, opaque, sealed envelopes.
Inadequate concealment methods include:
- Using an open random allocation schedule (e.g. a list of random numbers).
- Assignment envelopes were used without appropriate safeguards (e.g. if envelopes were unsealed or non-opaque or not sequentially numbered)
- Alternation or rotation
- Date of birth
- Case record number
5. Participants and investigators need to be masked
Both the participants and investigators need to be “blind,” or “masked,” to the intervention that was received. Blinding means that neither the participants nor the investigators know which group the participants have been placed into, which reduces the risk that knowledge of the intervention status could affect outcome measurement. In some trials it is not possible or ethical to blind participants and/or the investigators. But the ideal scenario is that neither know the intervention status because biases and beliefs can influence the interpretation of how outcomes are perceived and measured. For example, if participants know which group they are in, knowledge that they are receiving the active intervention may lead them to overestimate self-reported outcomes, or those in the placebo control group to underestimate outcomes. Similarly, investigators may over or underestimate outcomes if they know participants’ group status. Blinding helps ensure that the outcomes are measured as objectively as possible.
6. Make sure all data is analyzed and reported
Ensure that data from all participants are analyzed and reported. If data from some participants are not included in the analysis, the results may be incorrectly estimated This leads to biased conclusions. If some participants drop out of the trial before it is completed (data from all participants that were enrolled in the trial could not be included) an intention-to-treat (ITT) analysis should be conducted. An ITT is where data from other participants in the trial are used in place of those that dropped out of the trial. An ITT analysis ensures the statistical power is maintained. Statistical power is vital to estimate true outcomes. Naïve per-protocol analyses (where participants who did not receive their assigned intervention are excluded) and as treated analyses [where participants are analyzed in groups according to the intervention that they received (e.g., if some participants switched groups rather than according to their assigned intervention] are inappropriate analysis methods.
Final thoughts
In addition to performing each of these steps when conducting a trial, all these components must be described in the final report. Trials are evaluated for bias after the research has been disseminated. Even if these components were conducted, if they have not been reported in the manuscript the research will be marked down and considered biased if any of these items are missing.
Cochrane Collaboration is a leading evidence-based medicine organization that conducts systematic-reviews and meta-analyses. Their risk of bias tools are considered the gold-standard for evaluating bias. The five components described above are all elements that must be addressed in Cochrane’s RCT evaluation tool (ROB 2).
While it is important to evaluate trials for bias after they have been conducted, as a gauge for how trustworthy the results are, the goal for the clinical research community should be to conduct better trials from the start so the end result is a trial unbiased by design.