Estimating the External Validity of Randomized Controlled Trials: A Comparison of Morbidity and Mortality Between an RCT and an Observational Study in Botswana


 Background: The external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the RCT apply to the non-trial population. External validity of RCTs is impacted by the representativeness of the population enrolled and differences in the standard of care between the trial and non-trial setting. In spite of these concerns that have been historically acknowledged, external validity of RCTs is difficult to study empirically given confounding due to time and population under study. Here, we compared the outcomes of mortality and hospitalization between an RCT and an observational study that concurrently enrolled HIV-exposed uninfected (HEU) newborns in Botswana. Methods: The Mpepu Study was a clinical trial that enrolled HEU newborns in Botswana to determine whether co-trimoxazole provided survival benefit. The Maikaelelo study was an observational study that enrolled HEU newborns in Botswana with telephone follow-up and no in-person visits. Mortality and morbidity were compared using survival analysis and treatment effects approaches. Cox-proportional hazards model were fit with time to death or time to first hospitalization and where the proportional hazards assumption was determined to be violated, interaction terms were generated between the study setting and time. The causal effect of study setting was obtained through the inverse probability weighted estimator. Results: In total, 4,010 infants were included; 1,306 were enrolled into Maikaelelo and 2,704 were enrolled into Mpepu. No significant differences in mortality were observed between the two settings (HR: 1.28, 95% CI: 0.76, 2.13), but RCT participants had a lower risk of hospitalization (HR: 0.72, 95% CI: 0.58, 0.89) that decreased with age. However, RCT participants had a higher risk of hospitalization within the first six months of life. The causal risk difference in hospitalizations attributable to the trial setting was -0.03 (95% CI: -0.05, -0.01). Conclusions: Children in an RCT with rigorous application of national standard of care guidelines experienced a significantly lower risk of hospitalization than children participating in an observational study that did not alter clinical care. Future research is needed to further investigate outcome disparities when real-world results fail to mirror those achieved in a clinical trial. Trial Registration: : The Mpepu Trial was funded by the U.S. National Institutes of Health (No. NCT01229761) and the Maikaelelo Study was funded primarily by the U.S. Centers for Disease Control and Prevention (32AI007433-21).


Background
Randomized controlled trials (RCTs) are often considered the highest grade of evidence in clinical research because the randomization process aims to balance both known and unknown confounders.1 In doing so, these trials seek to measure the true causal effect of an intervention on a given outcome. 2 Historically, much of the literature discusses "internal" aspects of trial execution that may introduce bias to the estimation of causal effect. 3 What is less discussed, however, is the external validity of clinical trials -that is, the extent to which the results of a clinical trial can be exported to a population of interest.4,5 There are various dimensions to the conduct of clinical trials that improve the exportability of their results to the real-world setting. The PRECIS-2 tool considers nine of these domains, each of which is scored on whether the trial answers questions about pragmatism under real-world conditions or e cacy under ideal conditions. 6 The concerns that the idealistic conduct of trials deviate from how interventions are applied in the real-world have led to a movement toward both pragmatic clinical trials and for implementation science. 7 Where the conduct of a clinical trial exceeds the standard of care available in real word settings, a pragmatic trial offers the opportunity to strengthen external validity. Alternatively, in resource limited settings where the standard of care is not consistently maintained in a health care system but is adhered to in a clinical trial as an ethical obligation to participants, implementation science provides the methodology to identify programmatic opportunities to ensure that ideal outcomes reported in trials are achieved in real world settings. The motivation to assess the external validity of clinical trials is di cult to study empirically. When it is studied, analyses are often exclusively focused on whether trial populations accurately re ected the non-trial population about which the trial sought to make inferences.8 Further, the robustness of these analyses is threatened by confounding due to time and population under study. Here, we conduct an empirical analysis of the external validity of a clinical trial with respect to three domains: eligibility, setting, and organization.9 By leveraging data from two concurrent studies, we captured the extent to which enrollment into the clinical trial setting affected both morbidity (hospitalization) and mortality compared to an observational setting that represented routinely accessed care in Botswana.

Methods
The data for this analysis originates from two studies in Botswana: the Mpepu Trial (Clinical Trial No. NCT01229761), a randomized clinical trial, and the Maikaelelo Study, an observational cohort study that captured data using telephone follow-up. In Additional File 1, we describe the relevant considerations of each study and its execution.

Methods -Dataset:
A combined dataset of children enrolled in the Mpepu Trial and the Maikaelelo Study was created.
2.1.1 Inclusion Criteria: Infants were included if they were HEU, as determined by the de nitions of the respective study. To be conservative in the approach, infants whose HIV status was unknown by the end of the study period were excluded.
2.1.2 Exclusion Criteria: Infants were excluded from the analysis if they died or were hospitalized prior to enrollment. Infants were also excluded from the analysis if maternal HIV treatment during pregnancy was missing. Infants from the Mpepu study who were not randomized (and therefore did not enroll into the trial) were also excluded. Infants who died or were hospitalized within 30 days of birth were excluded from the analysis to ensure comparability between trials given their different enrollment strategies.
Finally, events among infants in Maikaelelo that occurred after 547 days or 18 months (the length of follow-up in Mpepu) were excluded from the analysis.

Clinical Site:
Given the non-overlapping clinical sites in Botswana from which these children were enrolled, an indicator variable was used to adjust for sites in large urban settings (Francistown and Gaborone) compared to sites in more rural or peri-urban settings (Maun, Ramotswa, Mochudi, Kanye, Lobatse, and Molepolole).

Socioeconomic Status:
Since both studies collected comparable socioeconomic data, a socioeconomic score was created ranging from 0 to 7, with 0 corresponding to lower socioeconomic status and 7 corresponding to a higher socioeconomic status. This score was a summation of the scores of maternal education (from none/primary to secondary to university), household access to electricity (from no access to access), source of water (from not piped into the home to piped into the home), and housing structure (from no stable housing, to informal housing, to mixed formal/informal, and to formal housing).

Outcomes:
Outcomes of interest were death and hospitalization as binary outcomes, as well as time to death and time to rst hospitalization.

Survival Analysis:
To understand the relationship between study type, time, and either of the outcomes of death or hospitalization, two Cox proportional hazards model were t modeling either time to death or time to rst hospitalization from 1-18 months, adjusted for the aforementioned covariates. Where the proportional hazards assumption was thought to be violated, an interaction term between the study setting variable and the analysis time was generated. An interaction term between the natural logarithm of analysis time and study setting was modeled using restricted cubic spline transformations to visually assess the changes in the hazard ratio over time.

Causal Effect Estimation:
The causal effect of study setting was obtained through an inverse probability-weighted estimator, and inferences were made with 100 replications of the bootstrapped standard errors. The results of this approach report two coe cients: the potential-outcome mean (POM) which is the risk of the outcome had all children been enrolled in Maikaelelo and the average treatment effect (ATE) which is the average difference in risk between the potential risk had all children been enrolled in Maikaelelo and the potential risk had all children been enrolled in Mpepu.

Sensitivity Analyses:
Sensitivity analyses were also conducted in which infants who died or were hospitalized within 30 days of enrollment into either study were excluded to determine whether these inferences held even after excluding the earlier effects of study setting.

Results
3.1 Baseline Demographics: Table 1 describes the characteristics of the infants who were HEU presented by study. In total, 4,010 infants were included in the analysis, 1,306 (32.57%) from Maikaelelo and 2,704 (67.43%) from Mpepu. The Mpepu trial enrolled more infants from large urban settings compared to the Maikaelelo study (59.17% vs. 46.86%, p < 0.001) and more mothers in Mpepu chose to breastfeed their infants (22.74% vs. 14.09%, p < 0.001). Mothers in Mpepu were more likely to be receiving 3-drug ART than no ARV strategy or ZDV only compared to mothers in Maikaelelo (86.21% vs. 67.08%, p < 0.001). On average, mothers in Mpepu had a higher socioeconomic status (4.39 vs. 4.26, p = 0.003) and a higher baseline CD4 count (540.62 cells/mL 3 vs. 448.08 cells/mL 3 , p < 0.001). No signi cant differences were observed between the two studies with regards to sex (p = 0.79), mean birthweight (p = 0.24) or low birthweight status (p = 0.22) of the children. After applying the exclusion criteria mentioned earlier, no infants used in the analysis had missing outcomes.

Mortality:
In the cox proportional hazards model tted for time to infant death (Table 2), and adjusted for the covariates of sex, clinical site, socioeconomic status, breastfeeding strategy, type of maternal HIV Page 7/12 treatment during pregnancy, and low birthweight, study setting was not associated with a signi cant change in the hazard of death during the follow-up period (HR: 1.28 95% CI: 0.76, 2.13).

Morbidity:
In the Cox proportional hazards model tted for time to rst hospitalization, and adjusted for type of maternal HIV treatment during pregnancy, sex, socioeconomic status, and infant feeding strategy, enrollment into Mpepu was associated with a time-xed hazard ratio of 0.72 (95% CI: 0.58, 0.89). An interaction term between analysis time in years and study setting was generated and tted with the original covariates. The interaction term in the re-tted model was associated with an HR of 0.42 (95% CI: 0.31, 0.58). A visual assessment of the change in HR over time suggested that the HR may initially be greater than 1 and over time decreases well below the null (Figure 1). The qualitative change in the HR occurs at approximately six months of age. After this, the point estimate for the HR departs signi cantly from the null and continues to trend downwards until the end of the analysis time.

Treatment Effects:
3.3.1 Mortality: The causal effect of the Mpepu trial on mortality was obtained through an inverse probability weighted estimator. Had everyone been enrolled in Maikaelelo, the average risk of death would have been 0.02 (95% CI: 0.01, 0.02). No difference in mortality was attributable to enrollment into Mpepu (ATE: 0.00 (95% CI: -0.00, 0.01).

Morbidity:
A similar approach was used to estimate the causal effect of enrollment into Mpepu on the risk of hospitalization. Had everyone been enrolled in Maikaelelo, the average risk of hospitalization would have been 0.12 (95% CI: 0.10, 0.13). The causal risk difference for hospitalization attributable to enrollment into Mpepu was -0.03 (95% CI: -0.06, -0.01), suggesting a decrease in the risk of hospitalization by approximately 29.77%.

Sensitivity Analysis:
When 43 infants who experienced outcomes within 30 days of enrollment were excluded, conclusions regarding the effect of study setting on mortality remain null while the effect of study setting on morbidity become even more dramatic. In this smaller population, the average treatment effect of the RCT on mortality was 0.00 (95% CI: -0.01, 0.01) while the average treatment effect on morbidity was -0.05 (95% CI: -0.07, -0.02), suggesting a decrease in the risk of hospitalization by approximately 39.45%.

Discussion
Our study quanti ed the protective effect of participation in a clinical trial with in-person follow-up compared with routinely accessed care among HEU children in Botswana. Although mortality did not differ, the risk of hospitalization decreased between 30-40% among children participating in the clinical trial.

Effects on Mortality:
There was no evidence of a protective effect on mortality from the clinical trial. Given the highly conservative nature of the exclusion criteria to ensure the validity of the inferences made, many infants who died in both settings were excluded and thus we had limited power to detect any meaningful differences in mortality between the trial and non-trial settings after these children were excluded.

Effects on Morbidity:
Both the primary and sensitivity analyses of morbidity suggested a strongly protective effect from receiving care in the clinical trial setting. The estimate of the causal reduction in the risk of hospitalization (29.77% in the primary analysis and 39.45% in the sensitivity analysis) remained robust, as did the time-varying effects of the trial setting. It is unlikely that a high early HR associated with the trial setting is driven by a biological mechanism given that children with a low-likelihood of survival were excluded from both studies. It is more likely that this early risk is driven by the intentional, physiciandirected hospitalization of study subjects in the RCT setting, a phenomenon that was not part of the observational setting, by design. After this time period, the HR continued to decrease suggesting that the long-term and involved care provided by the trial setting protected its subjects from excess hospitalization between six and eighteen months of life.
The most plausible interpretation of our results, therefore, is that physician availability in the trial setting allowed better identi cation of at-risk children, and in some cases led to early referral of those who were acutely or chronically ill, had missed vaccinations, or were failing to meet growth or developmental standards for age. This degree of management and care was ultimately associated with a lower overall risk of hospitalization through 18 months. This phenomenon, which re ects an outcome that could be achieved in health care settings when standards of care are uniformly and consistently applied, highlights the need for implementation science research to identify barriers to implementation of standards of care in real-world health care settings.

Strengths and Limitations:
Strengths of our study included the use of two existing cohorts in comparable locations in Botswana that were followed during an overlapping calendar period and captured similar health outcomes, allowing us to largely isolate the difference between in-person (clinical trial) and telephone (available care) follow-up. The studies had additional favorable characteristics, including the fact that the Mpepu clinical trial showed no difference between CTX vs. placebo (allowing us to use data from both arms), and the fact that the Maikaelelo telephone follow-up was deliberately intended to have no impact on clinical care delivery in its design and implementation. Both studies had extremely high rates of follow-up and completeness of data, which allowed meaningful assessment of morbidity and mortality through 18 months. Only 5% of children were lost to follow up in the Mpepu trial and vital statuses at 24 months were missing for only 0.5% of children in Maikaelelo.
The results of this study are speci c to a high HIV burden, low-resource context. The protective effects of the trial setting here are relative to routine care in Botswana. These differences may not be observed in high-income settings, where the difference in care offered in a clinical trial may not differ appreciably from care offered in the routine delivery of health care in a non-study setting. Additional limitations include the different study sites used for each trial, which could have led to several types of bias if there were differences in care or disease risk by site. However, the sites for both studies were similar in size and in resource availability and we are not aware of differences in health metrics that would have been likely.
There were also some baseline differences between trial participants. These included slightly higher access to ART in pregnancy among the mothers of enrolled children in the Mpepu Study, which could have had some unmeasured or residual confounding of health outcomes for children later in life. x Mothers in Mpepu were of a slightly higher socioeconomic status than mothers in Maikaelelo, but an average difference of less than 0.2 points on a 7-point SES scale is unlikely to suggest a substantive difference in SES between the trial and non-trial populations. To partially alleviate the concern of overrepresentation of healthier children in Mpepu infants from both studies who died within 30 days of birth were excluded from the analysis. Finally, the results of the sensitivity analysis conducted suggested that the protective effects of trial care may occur further into enrollment, and in the case of the study population, further into early life.

Conclusion
Our results provide empirical evidence to help quantify the difference in health outcomes between clinical trials and routinely accessed care in resource constrained settings. For children who are HEU, we estimate that there was nearly a 30% reduction in hospitalization by participating in a clinical trial. Outcomes in the non-trial setting may fall short of what is achievable for to a myriad of reasons, including limited health personnel, inadequate training of staff, supply chain shortages, or barriers to patient accessing care. While pragmatic trials are warranted to ensure external validity of study ndings, it is equally important to conduct implementation science research when results of clinical trials are not routinely observed in realworld settings.