Background: Issues with specification of margins and analytic methods can potentially bias results towards the alternative in randomized noninferiority pragmatic trials. To investigate this potential for bias we conducted a targeted search of the medical literature to examine how noninferiority pragmatic trials address these issues.
Methods: An Ovid MEDLINE database search was performed that identified any publications in New England Journal of Medicine, Journal of the American Medical Association, Lancet, or British Medical Journal published between 2015 and 2021 (inclusive) that included the words “pragmatic” or “comparative effectiveness” as well as “noninferiority” or “non-inferiority” in a multi-purpose search. Our search identified 14 potential trials of which 12 met our inclusion criteria.
Results: Of the 12 randomized pragmatic noninferiority trials, 11 were individually randomized trials and one was a cluster randomized trial. Ten of the 11 individually randomized trials met the criteria established for noninferiority as did the one cluster randomized trial. Noninferiority margins were prespecified for all the trials. The majority of margins (6) were based on either minimum clinically important differences, clinical experts, or consensus, while others were based on sample size, empirical data, or clinical decision. For two trials, no justification for the margin was provided. All trials conducted intent to treat or modified intent to treat analyses along with per protocol analyses and reached similar conclusions. Only two trials included all randomized participants in the primary analysis, one of which used multiple imputation to impute missing data. The percentage of participants excluded from the primary analysis ranged from about 2% to nearly 30% and sometimes differed between treatment arms.
Conclusions: Specification of margins and methods of analysis require careful consideration to prevent bias towards the alternative in noninferiority trials. Much of the guidance on these two issues has been developed for a regulatory environment and not for pragmatic noninferiority trials. Since many pragmatic trials generally follow the PRECIS criteria of little or no monitoring of participant or practitioner adherence, it affects separation of treatments which in turn affects both the setting of margins and analysis. More recent developments on estimands can address the latter issue.