How Much Is the Lack of Retention Evidence Costing Trial Teams in Ireland and the United Kingdom?

Abstract


Background
Randomised trials can be no better than the data they collect. If retention has been poor, in other words, data are missing for many participants, then the usefulness of the trial starts to come into question.
Patchy datasets can lead to a trial becoming underpowered for its primary outcome and very underpowered for its secondaries, which were anyway probably underpowered. Participants who do not provide data may differ to those who remain (1-3), making interpretation harder. It might be possible to overturn trial conclusions by simply imagining that the results from missing participants had gone against those conclusions (4). In short, potential users of the trial results now have doubts and doubt undermines trials.
Poor retention is a major cause of research waste as it can delay the implementation (or removal) of healthcare interventions (5). Poor retention may also increase trials costs (5,6). A recent prioritisation exercise (PRioRiTy 2) identi ed 20 priority unanswered research questions for research in trial retention, including questions around what motivates participants to stay involved, how to provide information and how what is done at recruitment might in uence retention (7).
Although there are plenty of unanswered research questions in trial retention, trial teams nevertheless have to use some form of retention strategy in their trials. In the United Kingdom (UK), Clinical Trials Units (CTUs) registered with the United Kingdom (UK) Clinical Research Network use many approaches and Kearney et al., asked them what these strategies were. Thirty-three (70%) CTUs responded and described a total of 61 strategies, some of which were used by most CTUs (8). While evidence is available for some strategies, none have compelling evidence of bene t (9).
The current study aimed to estimate how much this lack of retention evidence might be costing trials in the UK and Ireland. To give focus, we chose to do this for the ten strategies used most often by UK CTUs (8).

Methods
The top ten most routinely used retention strategies used by UK CTUs are shown in Table 1, along with the percentage of CTUs that use this intervention routinely (8) and the evidence of bene t provided by the most recent Cochrane Systematic Review (9).
To estimate the cost of the retention strategies, we made assumptions as to how each of these strategies would be implemented and the costs involved. We called this our costing model for each strategy. We contacted experienced clinical trial professionals such as trial managers, clinical research nurses, and professionals working in clinical research facilities in the UK and Ireland for information to inform our costing models. We identi ed these individuals through workplace inquiries and personal knowledge of suitable personnel to answer speci c costing queries.
We applied our costing models to each of the ten strategies in two ways. First, we created a hypothetical trial scenario (Table 2). We made assumptions about how the retention strategies would most likely be conducted and then calculated and applied the costs of running each of the ten retention strategies in both Ireland and the UK. The costing models for some strategies differed slightly between Ireland and the UK due to differences in the responsibilities held by staff members involved in running clinical trials and differences in how some retention strategies are conducted in each country (See Additional File 1 for full details).
Secondly, we chose three trial protocols published in the journal Trials between 2020 and 2016. We chose these three for convenience reasons: they t well into our costing model well, represented trials of various sizes and the trials had different characteristics. We used these to estimate the cost associated with each retention strategy in a "real-life" randomised controlled trial. We applied our costing model for the previously identi ed top 10 retention strategies to each of the three randomised controlled trials regardless of whether the trial protocol stated that these retention methods were used or not. An example costing model for both a hypothetical trial and published trial protocol (10), with costs and assumptions, is shown in Table 3. A full description of the costing models and the assumptions we made to create them, together with details of the three trials selected from the Trials journal are outlined in Additional File 1.
In addition to the ten most routinely used retention strategies, we further elaborated and included additional actions that are likely to be carried out by trial units when trying to retain participants that were not speci cally outlined by Kearney et al.,(8). These assumptions re ect actions that are likely to be conducted by trial teams during the implementation of the listed interventions. For example, under the intervention "inclusion of pre-paid envelopes (questionnaires)" along with sending out pre-paid envelopes to enhance questionnaire response we also assumed a reminder schedule would be sent out to 30% of participants who initially did not return the questionnaire. A full list of the additional assumptions can be found in Additional File 1.
Where there was evidence of effectiveness of the retention strategy we calculated the cost per participant retained. For example for the ease of calculation, we calculated the cost per participant retained by using pre-paid return envelopes based on a 4% bene t (9). In the Irish hypothetical trial scenario 4% of 500 participants is 20. The total cost of "inclusion of pre-paid envelopes (questionnaires)" was €942.50. Therefore the cost per participant retained was €47.13 per 20 additional retained participants.
All our calculations were done within Microsoft Excel. Additional File 2 contains our Excel spreadsheet with our cost calculations. For ease of comparison all our costs are presented in Table 4 in both EUR and GBP based on exchange rates of 1 GBP = 1.16279 EUR, 1 EUR = 0.860001 GBP, 1 USD = 0.843802 EUR and 1 USD = 0.725647 GBP, taken from Xe Currency Converter -LIVE Foreign Exchange Rates on the 7 th of September 2021.

Results
We discussed the top ten most routinely used trial retention strategies in the UK, we made assumptions as to how each of these strategies would be implemented and the costs involved. We applied this costing model to our hypothetical trial scenario, (1-year trial, 10 sites, 500 participants, 3 trial visits, 1 questionnaire sent out) in the UK and Ireland, and found that the strategies ranged from very cheap to very expensive to implement. The cheapest strategy to implement in both countries would be "telephone reminders for questionnaire response" costing €423.75 (£364.43) and €568.62 (£489.01) respectively and the most expensive strategy would be "a timeline of participant visits for sites" with integrated participant reminders costing €68,021.75 (£58,498.77) in Ireland, but "data collection scheduled with routine care" in the UK (€15,697.67 (£13,500)).
For the "real life" trials, "telephone reminders for questionnaire response" would be the cheapest strategy in the CINNAMON trial (€246.00 (£211.56)). For the MAMI and MOON trial "targeted recruitment of sites/GPs" would be the cheapest strategy in these two single centre trials costing (€30 (£25.80)) and €35.35 (£30.40) respectively. The most expensive strategy in the MAMI and the CINNAMON trial would be "a timeline of participant visits for sites"(with integrated participant reminders) costing €2,945.34 (£2,533.00) and €108,471.99 (£93,282.99) respectively. For the MOON trial the costliest intervention for the trial team to conduct would be "data collection scheduled with routine care" costing €4,834.88 (£4,158).
Newsletters were identi ed as the most routinely used retention strategy by CTUs in the UK (8). The cost of sending newsletters would be one of the cheaper retention strategy options ranging from €734.51 (£631.68) to €1,990 (£1,711.40) for manually posting the newsletters, and €525.93 (£452.30) to €1,036.39 (£891.27) to electronically send newsletters. However the most up-to-date evidence suggests there is no retention bene t of this strategy "RD = -0% (95% CI -4-3%); GRADE: very low" (9). This costing model has shown that "a timeline of participant visits for sites" would be a cheap option if only site reminders were used (€101.34 (£87.15) -€1,119.50 (£962.77)) but if a participant reminder schedule was also integrated into this strategies it would be one of the most expensive strategies for trial teams to implement (€1,418.44 (£1,219.86) -€108,471.99 (£93,282.99)). Regardless of which option is used neither strategy has available evidence to support its use (9) however is the second most routinely used retention strategy identi ed by Kearney et al.,(8).
The retention strategy with the best available evidence is "inclusion of pre-paid envelopes" for questionnaire return. The cost of this retention strategy ranges from €93.44 (£80.36) to €942.50 (£810.55), one of the cheapest retention strategies trial teams can avail of. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal follow-up" all combined may lead to a 4% retention bene t "RD= 4% (95% CI -0-9%)". However, the evidence was based on three low quality studies (n=1543), and the single study of pre-paid return envelopes itself, did not nd a positive effect on retention (9).

Discussion
Our ndings show that the evidence available to support the ten most-used trial retention interventions by CTUs in the United Kingdom is weak or lacking entirely but that the cost of using them can be very large.
The most routinely used intervention outlined by Kearney et al., (8) is "newsletters", this strategy was found to be one of the cheaper retention methods particularly emailing newsletters, (€525.93 (£452.30) to €1,036.39 (£891.27)). One of the cheapest interventions across all the trials would be "telephone reminders for questionnaire response" costing between €34.58 (£29.74) and €568.62 (£489.01). We are able to say that these interventions would be cheap but more evidence is needed to show that it is also effective at retaining trial participants.
The second most routinely used retention strategy outlined by Kearney et al.,(8) is "a timeline of participant visits for sites". The site reminder schedule alone would be cheap costing between €101.34 (£87.15) and €1,119.50 (£962.77). Integrating a participant reminder schedule would signi cantly increase the costs (€1,418.44 (£1,219.86) -€108,471.99 (£93,282.99)). Similarly, "routine site visits by CTU staff" (€777.67 (£668.80) -€14,753.48 (£12,688)), "investigator meetings face-to-face" (€777.67 (£668.80) -€14,753.48 (£12,688)), and "data collection scheduled with routine care" (€900 (£774.00) -€32,503.25 (£27,951.92)) would also be expensive to implement yet none of these have compelling evidence demonstrating that they are effective at retaining trial participants (9). They may be very effective. The point is that we cannot say with any certainty whether they work or not, and therefore substantial amounts of money and other resources are potentially being invested into strategies that lead to no improvement in retention.

Strengths and limitations
We acknowledge that a limitation of this study is that we have had to make assumptions to calculate our cost estimates and these may not be truly representative, or the assumptions made may not be accurate depending on how trials are run, especially those outside of Ireland and the United Kingdom. However, to help to address this limitation, we have made the costing spreadsheet available as an additional le, which means readers can modify it to suit their own trial.
One of the strengths of this study is that regardless of the costs, it highlights the lack of evidence for routinely-used trial retention strategies. Even if our estimates are very wrong, no intervention costs nothing and if there is weak or no evidence in support of the intervention, we should pause and consider what we want to do. If trialists go ahead and use the strategy, we think at least some of them should use SWATs (Study Within a Trial) or other research design to investigate the impact of the strategy on retention. The combination of routine use of a strategy to support retention and a lack of evidence that the strategy actually improves retention is a recipe for research waste.
Recommendations for future research This paper highlights the need for further research into the effects of trial retention strategies. The cost of some of the interventions that are currently routinely used are signi cant, and so is the lack evidence to support their use. We recommend the wider use of SWATs to evaluate the effects of retention interventions used in clinical trials to avoid persistent and widespread research waste. Replication of evaluations will add to the existing evidence to support/not support the use of these interventions.
We also think it would be useful for trial teams to include the costs of running trial retention strategies in their trial publications. Communicating the costs of retention strategies can be helpful to other trial teams to estimate budgets required for implementing similar strategies. A better idea of costs will allow for better 'cost-per-participant-retained' calculations, which in turn will give trial teams another way to compare retention strategies when making choices about which to use in their trial.

Conclusions
Without evidence regarding the effectiveness of trial retention strategies, trial teams will continue to put substantial amounts of money into strategies that potentially have no bene cial impact on participant retention. More evaluation of the effectiveness and cost of trial retention strategies is needed to avoid widespread use of strategies that are both expensive and ineffective.  "Telephone reminders compared to usual follow-up 1 may result in little or no difference in retention, evidence from one study (12) (RD =-1% (95% -18-15%) GRADE low, (-2 levels: imprecision-single study, n = 127; wide CI crossing RD = 0))" 2 5 Data collection scheduled with routine care N=18, 55% No evidence Notes: GRADE, grades of evidence; Low certainty: the con dence in the effect estimate is limited; the true effect may be substantially different from the estimated effect. Very low certainty: very little con dence in the effect estimate; the true effect is likely to be substantially different from the estimate of the effect (9). 1 From this single study "usual follow-up" is as follows; "We followed-up participants by any of the means they agreed to at the start of the trial, including post, e-mail, and telephone calls to mobile, home, or work numbers [11]. We used all the effective evidence-based methods that were feasible to introduce into the procedures of the trial [12], as identi ed in the systematic reviews by Edwards et al. and Hoile et al. [12,13]. These included monetary incentives, posting correspondence by recorded delivery, pre-noti cation, follow-up contact, unconditional advance cash incentives, short, concise questionnaires, duplicate questionnaires sent at repeat follow-up attempts, mentioning that commitment to the trial implied an obligation to respond, mention of university sponsorship, prepaid return envelopes with stamps, an assurance of con dentiality, and rst-class outward mailing" (12). 2 We present all evidence regarding telephone reminders however we chose to select telephone reminders compared to usual follow up in Table 4 to calculate cost per participant retained as we believe it is the most relevant comparison for trial teams. Notes: GRADE, grades of evidence; Low certainty: the con dence in the effect estimate is limited; the true effect may be substantially different from the estimated effect. Very low certainty: very little con dence in the effect estimate; the true effect is likely to be substantially different from the estimate of the effect (9). 1 From this single study "usual follow-up" is as follows; "We followed-up participants by any of the means they agreed to at the start of the trial, including post, e-mail, and telephone calls to mobile, home, or work numbers [11]. We used all the effective evidence-based methods that were feasible to introduce into the procedures of the trial [12], as identi ed in the systematic reviews by Edwards et al. and Hoile et al. [12,13]. These included monetary incentives, posting correspondence by recorded delivery, pre-noti cation, follow-up contact, unconditional advance cash incentives, short, concise questionnaires, duplicate questionnaires sent at repeat follow-up attempts, mentioning that commitment to the trial implied an obligation to respond, mention of university sponsorship, prepaid return envelopes with stamps, an assurance of con dentiality, and rst-class outward mailing" (12). 2 We present all evidence regarding telephone reminders however we chose to select telephone reminders compared to usual follow up in Table 4 to calculate cost per participant retained as we believe it is the most relevant comparison for trial teams.  Table 3 Assumptions and costings for "Newsletters" -Hypothetical trial Ireland and the MAMI trial (10).

Hypothetical trial (Ireland) Intervention
Assumptions made Newsletter → 2 newsletters sent out over the 1-year trial period.
→ 5 hours to develop the newsletter.
→ 1 hours work to electronically send out 500 newsletters.   1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.

Strategy Cost
Effect estimate "The evidence is very uncertain about the effect on retention of including a newsletter compared to no newsletter: RD = -0% (95% CI -4-3%); GRADE: very low"  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.  1 Italic underlined font indicates the real trial information that we used e.g., number of trial participants, number of trial sites, number of trial visits and any retention methods used within the trials e.g., MOON trial sent out 2 questionnaires to participants. The non-italic-underlined font which indicates the assumptions we applied. A full description of the trial characteristics and any retention activities conducted by the real-life trials are documented in Additional File 1. 2 Evidence from the most recent Cochrane Systematic Review (9). 3 We assumed bene t was applied across the whole response to all cycles. The Cochrane review found that return postage which included "preaddressed second class stamped envelope", "high priority stamp to the mailing" and "personalised postal followup" all combined likely lead to a 4% bene t, so it is likely that pre-paid envelopes on their own may not provide a 4% bene t on retention. For the ease of calculation, we calculated the cost per participant retained by using pre-paid envelopes based on a 4% bene t. 4 We chose to select telephone reminders compared to usual follow up compared to postal follow up in as we believe it is the most relevant comparison for trial teams.