Yunyun Jiang

Wenle Zhao

Valerie Durkalski-Mauldin

Figure 1

Figure 2

Figure 3

<p>Bayesian Response Adaptive Randomization in Clinical Trials With a Time-trend: Application to a Real Trial</p>

<p>This preprint is available for <a href='/article/rs-35671/latest.pdf' target='_blank'>download as a PDF</a>.</p>

BMC Medical Research Methodology

CC BY 4.0

<h2>Background</h2> <p>In clinical trials with a large sample size, time-trend in response rates can affect the performance of Bayesian response adaptive randomization (BRAR).</p><h2>Methods</h2> <p>To evaluate this impact, we utilize data from a previously completed randomized controlled trial that used a fixed 1:1 allocation. Subject response data from this study demonstrate a clear time-trend in the control group, but not in the treatment group. In this simulation study, we re-assign patients to treatment groups based on a BRAR algorithm, to examine the performance of BRAR as measured by the treatment effect estimation, the probability of early stopping, and the shift in adaptive allocation.</p><h2>Results</h2> <p>Results from specific simulated study scenario show that in the presence of a time-trend, the timing of the first interim analysis is critical for the efficacy/futility decision making. Compared with fixed equal allocation, BRAR results in a higher probability of premature early stopping when time-trend effect exists. The magnitude of such impact varies among different BRAR algorithms.</p><h2>Conclusions</h2> <p>Influential factors such as time trend, should be considered when planning the implementation of BRAR in large trials.</p><h2>Trial Registration:</h2> <p>URL: <span class="ExternalRef"><span class="RefSource">https://www.clinicaltrials.gov</span></span>. Unique identifier: NCT00235495. Registered on October 10, 2005.</p>

Bayesian Response Adaptive Randomization in Clinical Trials With a Time-trend: Application to a Real Trial

Recommendation: Major revisions required
Form responses:
---

Comments to Author:
---
Time trends in adaptive randomization (and platform trials) are timely. It is clear from the literature that time trends can adversely effect treatment effect estimates and increase testing error rates. Realistic examples of time trends as provided by the authors are valuable. However, I have several concerns and suggestions with the current example.

1) The ALIAS trial present a particularly challenging form of time trend. Time trends are often discussed in terms of an additive effect to both arms over time, while ALIAS appears to have an interaction. This creates interpretation issues that go beyond the adaptive randomization piece. If their truly is an interaction, what does the treatment effect even mean? If the control arm is truly worse at the beginning of the study we reject the null at the first interim, have we committed a type 1 error given the treatmnet was truly effective there? If the study doesn't even generalize internally, how can it generalize externally? In short, it's not clear this kind of time trend has a correct answer. It would be good to have a fuller discussion of these issues, but it's unclear there is a definitive answer.

2) A "redesign" paper such as this or READAPT has advantages and disadvantages. An advantage is that it shows a method or issues in the context of a real example, allowing practitioners a concrete example to work with and understand. However, the theoretical or simulation derived behavior of trial operating characteristics would seem to be the final word on whether a method is good or bad. In the case of a promising methodology, I can see the evolution from "this works in theory (or simulation)" to "here is how it would work in a practical example" to gain acceptance to "let's use this for real in a future example". Here the theoretical/simulation properties of the method indicate unadjusted time trends break RAR, so the practical example seems less valuable.

3) It would be valuable to show how the time trend breaks RAR, but also show how to correct this in a practical example. For example, one could formulate a model which bins time every N patients (every 30 is used in the graphs but the bins could be wider). With discrete parameters (e.g. rate = effect for arm + effect for bin) you can eliminate the majority of the bias if the time trend has no interaction, but again unclear of the interpretation of "treatment effect" when there is an interaction. This might require a different example with an additive, rather than interactive, time trend.

4) There are several groups working on time trend models beyond linearity in the presence of an additive time trend. In unclear how to model an interactive time trend. I've seen a generalized additive model recently using a spline over time, but only at a conference not in print. Some of the platform trials such as ISPY2 use a normal dynamic linear model to smooth over time.* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **No**
* Declaration of competing interests: **I declare that I have no competing interests**
* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please do not publish my name with my report. (default)**
* Is the study design appropriate to answer the research question (including the use of appropriate controls), and are the conclusions supported by the evidence presented?: **Yes**
* Are the methods sufficiently described to allow the study to be repeated?: **Yes**
* Is the use of statistics and treatment of uncertainties appropriate?: **Yes**
* Is the presentation of the work clear?: **Yes**
* Are the images in this manuscript (including electrophoretic gels and blots) free from apparent manipulation?: **Yes**


Recommendation: Major revisions required
Form responses:
---

Comments to Author:
---
This is a very interesting paper. The topic is highly relevant to those implementing novel adaptive trial designs in practice and the exercise of using real data to evaluate a methodological design deserves a lot of credit in itself. However, I do think most of the conclusions the authors are drawing in the current version of the paper are not new nor bring novel perspectives to a well known issue. This seems detrimental to a case in favour  of using the real data to evaluate trial design. Were these simulations actually needed to make the conclusions stated in the paper as to the use of BRAR? What is learnt from this exercise that could not be learnt from more theoretical work? In the comments to the author you will see that many of the work's conclusions appear almost exactly in the same words as in this manuscript. Some of these concerns may be addressed by better framing and writing of the paper but there are some important omissions in the references list of the relevant work which are a major concern. For example, the main conclusion of the abstract is the same one as in the Discussion of the Villar et al (2018) provided below, where it reads "therefore, an important recommendation of this work is that in the design stage of a clinical trial, when a RAR procedure is being considered, a similar detailed evaluation of type I error inflation and bias as done in this paper is performed to choose a suitable RAR procedure for the trial at hand."

Despite the above I think this work is valuable and could be properly modified to be published. It does require a genuine case for what it is added by it to what is already known. Please see below a list of detailed comments to help improve the paper towards publication.

Detailed comments:

- The authors claim that time trends can be an issue in clinical trials using BRAR with a large sample size. I would disagree with the fact that is the size of the trial that is a source of concern but the length in time that allows for the trend issue to lurk in. Trials that last several years, irrespective of their size, might see issues like those pointed out in this draft of a control arm evolving over time. The temporal aspect of the ALIAS trial is not well described. How long did this trial last? How separate in time are the interim analysis? A more detailed discussion of this aspect is much needed and adds a lot to the practical side of implementing a BRAR design. The timing of the interims is shown as important but this is tied to patient numbers where this can correspond to very different points in times depending on the primary endpoint of a study and the length of the follow up.

-  The conclusion of the paper is very poor. "Influential factors such as time trend, should be considered when planning the implementation of BRAR in large trials." This is not new and we did not need the simulations provided to know this. Many of the references cited do warn about this. If these are the main conclusions, then I do to see the added value of the simulations done. I am convinced this exercise is of value so I would encourage the authors to reflect on them more with this in mind.

- Time trends have been described in papers cited in this work (Simon & Simon, 2011) and in the reference below that has been omitted. There is an important distinction between a trend that equally affects all arms (this creates issues for RAR only and not traditional equally randomised trials) and trends that affect a subset of the arms in a trial. This should be discussed to inform the reader. Villar, Sofía S., Jack Bowden, and James Wason. "Response‐adaptive designs for binary responses: How to offer patient benefit while being robust to time trends?." Pharmaceutical statistics 17.2 (2018): 182-197.

- "However, the response time-trend has rarely been examined in trial practice"  (Background section). See Villar et al where it says: "Further research is also needed to assess the potential size of time trends through careful reanalysis of previous trial data (as Karrison et al do with data from Kalish and Begg)."

- A major limitation of the conclusions in the paper that is not discussed at all is the fact that two-armed trials and multi-armed trials may very well be substantially differently affected by a trend. This is a point made in the Villar et al (2018) paper.

- The authors mention they explore potential solutions to the problem. I am not convinced this was sufficiently or convincingly done. Simon & Simon (2011) discuss the use of randomisation based test as a solution robust to all types of time trends in a two arm trial. The authors do not even discuss this as a solution. What solutions do they propose? Why is not this a conclusion in the abstract?

- Methods. Thompson et al, who is et al here. The Thompson paper is a single author paper, please correct that.

- Methods: An important limitation of the work is that a very limited family of BRAR is considered. As discussed in Villar et al (2018) and more recently in Robertson et al (2020) this can bias results strongly as RAR from a same class are likely to be equally affected by trend (or not). There is a large class of RAR algorithms that are robust to this issue. However, patient benefit and efficiencies might be reduced for those creating difficult trade-offs. This is not discussed at all. Robertson, David S., et al. "Response-adaptive randomization in clinical trials: from myths to practical considerations." arXiv preprint arXiv:2005.00564 (2020).

- Where exactly was the idea " To reduce the allocation ratio variability, the square root transformation is employed." first proposed? Can this be referenced?

- "Unexpectedly, the response rate in the control arm increased over time while the response rate in the treatment arm remained stable across the trial." This deserves a bit more discussion. The control is saline as far as I understand. Improvement in it over time could be a shift in the population from worse prognosis early on in the trial to better prognosis for those recruited at the end. How long did the trial recruit for and how variable was the recruitment process? How could this impact on the observed trend? This is where the value of real data comes from and the authors could exploit this much more than what they have done.

- There are many design decisions that impact conclusions not well justified. Why 4 interims only? Why a burn in phase of that size and why not having an interim just after the burn-in phase (surely the data must be looked at to adapt the randomisation ratios, so I do not see why not allow for that).

- Early stopping is a distinct form of adaptation than BRAR. The authors are using it as if they were one unique thing. There are many BRAR simulations that do not include the early stopping possibility to isolate the effects of the RAR part on operating characteristics. How would the simulations look like if the trial would not stop?

- "Timing of the first interim analysis after the burn in period is critical when implementing BRAR, particularly in the presence of a time-trend." I would say that the burn in size and the timing of the first interim do interact on this effect significantly. How was the burn-in decided and why? Perhaps, a smaller burn-in would make things more variable? What is the variability of the sample sizes per arm?

- The re-design strategy proposed in this article is relatively novel. Yes, and this is where I see the largest value of the work but the conclusions around it are not compelling enough to convince there is added value beyond the work existing in the literature.

- "Some argue that the impact of time-trend is quite minimal to the adaptive designs if only mild trend effect exists. [11] However the definition or threshold for what constitutes a 'mild' trend is difficult to determine. " Again, this was pointed out in Villar et al (2018) where it reads: Further research is also needed to assess the potential size of time trends through careful reanalysis of previous trial data (as Karrison et al do with data from Kalish and Begg). Our results suggest that even in the case of a large time trend being a realistic concern, there are still some RAR rules that, both for the 2‐armed and the multi‐armed case, remain largely unaffected in all the cases we have considered. Of course, these rules offer increased patient benefit properties when compared to a traditional CR design but reduced when compared to the patient benefit‐oriented RAR rules.
If you see the simulations provided in that paper, some idea of what level of drift could start to be worrying for type I error  inflation in a binary endpoint case was suggested.

* Publons Reviewer Recognition. Springer Nature can send verification of this review directly to Publons (a subsidiary of Clarivate Analytics). If you would like to take advantage of this service, please click on the “Yes” option below. Your name, email address, title of the reviewed manuscript, name of the journal, and date of your review submission (the “Review Data”) will then be transmitted to Publons upon publication of the manuscript. If you have already registered at Publons, they will notify you of the receipt of this review and update your profile as per your settings and their policy. If you are not registered with Publons, you will receive an email from them asking you to register in order for them to be able to recognize your review on your new profile page. Publons may use the Review Data to generate derivative metadata for the benefit of Publons and you as a reviewer, carefully considering the sensitivity of such information. For example, Publons may verify your record as a reviewer by updating your profile published on its webservice if you have registered for such service or help editors to identify candidate reviewers. Please find the details of processing in Publons’ privacy policy https://publons.com/about/terms: **Yes**
* Declaration of competing interests: **I have no conflict of interest to declare**
* Reviewer Publication Consent. I agree for my report to be made available under an Open Access Creative Commons CC-BY License (http://creativecommons.org/licenses/by/4.0) if this manuscript is accepted for publication. Any comments that I do not wish to be included in the published report have been included as confidential comments to the editor, which will not be published.: **I agree to the terms of the CC-BY 4.0 license; please do not publish my name with my report. (default)**
* Is the study design appropriate to answer the research question (including the use of appropriate controls), and are the conclusions supported by the evidence presented?: **No**
* Are the methods sufficiently described to allow the study to be repeated?: **Yes**
* Is the use of statistics and treatment of uncertainties appropriate?: **Yes**
* Is the presentation of the work clear?: **No**
* Are the images in this manuscript (including electrophoretic gels and blots) free from apparent manipulation?: **Yes**


Recommendation: Reviewer's comments unavailable pending editorial decision


Bayesian Response Adaptive Randomization in Clinical Trials With a Time-trend: Application to a Real Trial

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Trial Registration:

Figures

Full Text

Status:

Version 1