Testing the Restrictiveness and Generalizability of Randomized Controlled Trial Population Sampling using a Failed Perioperative Beta-Blocker Trial

The limited applicability of evidence from randomized controlled trials in real word practice is considered a potential bottleneck for evidence-based practice but rarely systematically assessed. Using our failure to recruit patients into a perioperative beta-blocker trial, we set out to analyse the restrictiveness and generalizability of trial eligibility criteria in a real-world cohort. We included adult patients scheduled for elective major non-cardiac surgery at an academic tertiary care facility who were screened for inclusion in a planned perioperative beta-blocker RCT, which was terminated due to recruitment failure. Primary outcome was the proportion of screened patients who matched the eligibility criteria of 36 published RCTs included in a large Cochrane meta-analysis on perioperative beta-blocker therapy. The PRECIS-2 tool was used to assess the pragmatism/explanatory level of each RCT.


Background
Well-conducted randomized controlled trials (RCTs) are the gold standard for medical evidence and the basis for clinical decision-making. The applicability of evidence from RCTs in real word practice, however, may be limited [1,2] but is rarely systematically assessed. Patients enrolled in RCTs are commonly highly selected using restrictive eligibility criteria to reduce subject variability and bias [3]. However, restrictive eligibility criteria may impair a study's generalizability.
Generalizability refers to the external validity of study inferences and represents the extent to which study results can be extrapolated to different circumstances and populations. The pragmatic-explanatory continuum indicator summary 2 (PRECIS-2) tool has been developed to support the assessment of the pragmatic or explanatory level of a given RCT [4]. This is essential if trial results are to be translated adequately into clinical practice but rarely assessed by investigators [5]. Systematic assessment of the applicability of current evidence from RCTs may allow clinicians for a better interpretation of RCT-evidence to real world practice.
The current study was driven by our inability to recruit patients into an RCT due to restrictive eligibility criteria. The RCT aimed to determine efficacy and safety of perioperative beta-blocker therapy in patients with a high risk of cardiovascular mortality.
In the trial's screening phase we noticed that the potentially eligible trial population was very small (only one patient out of more than 2200 patients screened could be enrolled within a three months period). The inability to recruit patients at one of the largest tertiary care centres in Europe raised the concern that the generalizability of beta-blocker trial populations in a real-world cohort of patients may be limited.
We aimed to test this hypothesis by systematically assessing the proportion of patients of the screening cohort who matched the eligibility criteria of 36 RCTs included in a large 4 Cochrane meta-analysis [6].

Study setting and patients
This prospective cohort study was conducted at the Department of Anaesthesia, Medical University of Vienna at the General Hospital of Vienna, Austria, one of the largest tertiary care facilities in Europe. We screened adult patients (>45 years) scheduled for elective major non-cardiac surgery for eligibility for a planned perioperative beta-blocker RCT

RCTs included in the Cochrane meta-analysis
A large Cochrane meta-analysis was used as reference for high-quality evidence on the 5 effect of perioperative beta-blockers in patients undergoing surgery [6]. Two authors (MT, CK) independently extracted inclusion and exclusion criteria of each RCT (n=36, Table 1) and systematically assessed the eligibility proportions of our screening cohort.

Statistical methods
Data are presented as mean ± standard deviation (SD) or median with the 25-75% interquartile range (IQR), as appropriate. We determined proportions of screened patients matching the eligibility criteria for each available RCT together with 95% Jeffreys intervals. To estimate the average proportion of matches across all studies we used Poisson regression with a robust cluster variance estimator to calculate 95% confidence intervals. The pragmatic-exploratory continuum PRECIS-2 tool was used to assess the pragmatic/explanatory level of each RCT. The PRECIS-2 tool gives an estimate of the pragmatism/explanatory level of a trial ranging from a minimum of 9 points (very explanatory study) to a maximum of 45 points (very pragmatically conducted study). It contains 9 domains, which address the most relevant features of a trial, and was developed to support researchers in planning trial designs and to evaluate the impact of design decisions on applicability [4,7,8]. The domains include eligibility criteria, recruitment, setting, organization, flexibility delivery, flexibility adherence, followup, primary outcome and primary analysis [8,9].
We plotted proportions of matches versus RCT's sample sizes and their PRECIS-2 scores and used regression analysis to quantify the association. We report p-values from the Wald test. Stata Statistical Software: Release 14, StataCorp. 2017 College Station, TX: StataCorp LLC was used for data analysis and Prism 8 for OS X Version 8.3.0 to draw figures. Generally, we considered a two-sided p-value <0.05 statistically significant.

Results
A total of 2,241 patients (mean age 52years ±20, 54% female (n=1,215)) were included for the assessment of trial-eligibility between October 2015 and January 2016 ( Table 2).
Only a small proportion of patients matched the eligibility criteria for each of the 36 RCTs ranging from 53% to 0%. 6.5% of patients (n=145) were eligible for all 36 RCTs (95% confidence interval 6.3 to 6.6, Figure 1) and <10% of patients were eligible for 31 studies.
For nine published studies not a single patient from our cohort met the eligibility criteria.
There was no major criterion causing trial-ineligibility of study patients.
The median PRECIS-2 score of the RCTs was 18 points (IQR 17 to 20), indicating that the majority of RCTs were quite explanatory, less pragmatic studies. Figure 2 shows the nine domains of the PRECIS-2 tool and the PRECIS-2 scores of four representative RCTs included in the Cochrane meta-analysis.
A higher PRECIS-2 score was associated with a higher proportion of matching patients (p<0.001) (Figure 3).
The individual study sample sizes were not related to trial-eligibility (p=0.84; Figure 4).

Discussion
This study was based on the inability to recruit patients into a perioperative beta-blocker trial at one of the largest tertiary care centres in Europe. The recruitment failure raised the concern that the generalizability of beta-blocker trial populations in a real-world cohort of patients may be limited. We analysed the restrictiveness and generalizability of the trial eligibility criteria of a large Cochrane meta-analysis on perioperative betablockers in patients undergoing non-cardiac surgery in a real-world cohort. The proportions of matching patients were related to the sample size and the PRECIS-2 score of each RCT to determine the patients' detail matching.
As expected, only a small number of patients matched the eligibility criteria for each of the 36 RCTs. The sample size of individual studies was not related to trial-eligibility. It may be questioned whether the clinical impact of a trial should mainly be based on its 7 sample size, as frequently observed in scientific conversations, or additionally include a tool to rate the pragmatic and explanatory characteristics of a clinical trial design. Using the PRECIS-2 tool we found that very pragmatically conducted studies, which are designed to be externally valid, are missing in the field of perioperative beta-blocker therapy. The focus on exploratory studies is a suitable explanation for the lack of applicability of available evidence to our screening cohort. Standardized assessment and reporting of the PRECIS-2 score or equivalent tools [10, 11] by trial investigators may support readers in appraising a trial's degree of pragmatism and help clinicians to estimate the applicability of study findings to their patients in clinical practice [12]. Unfortunately, sufficient details are rarely provided. Real world evidence, however, is defined by the degree of pragmatism. Pragmatic studies are designed to represent routine practice care. Even the most effective treatment is useless if it does not work in the normal clinical setting.
Limitations: Although we identified an evidence gap for a clinically relevant topic potentially limiting the generalizability and applicability of trial results, we cannot rule out that trial results are also applicable to patients not included in the above trials. Moreover, looking at one example topic, we cannot generalize these findings to other fields, despite it is obvious that this may apply to all clinical research to some extent.

Conclusion
In conclusion we found that trial eligibility criteria in perioperative beta-blocker therapy trials are overly restrictive and not generalizable to a real-world population. Despite the availability of high-quality evidence from a large Cochrane meta-analysis, the applicability of results from perioperative beta-blocker trials may be limited. This may be partly explained by a lack of pragmatic studies in this field. Systematic assessment and reporting of the applicability of trial results may allow clinicians for a better interpretation of data generalizability. 8

Declarations Ethics approval and consent to participate
The study was approved by the Ethics Committee of the Medical University of Vienna and conducted in accordance with Helsinki Declarations.

Consent for publication
No individual patient data is reported that would require consent to publish fom patients.

Availability of data and materials
The datasets analyzed during the current study are available from the corresponding author on reasonable request

Competing interests
The authors declare that they have no competing interests