The NUDGE Trial Pragmatic Trial to Enhance Medication Adherence: study protocol for a randomized controlled trial

Abstract

Background Nearly half of patients do not take their cardiovascular medications as prescribed, resulting in increased morbidity, mortality, and healthcare costs. Mobile and digital technologies for health promotion and disease self-management offer an opportunity to adapt behavioral “nudges” using ubiquitous mobile phone technology to facilitate medication adherence. The Nudge pragmatic clinical trial uses population-level pharmacy data to deliver nudges via mobile phone text messaging and an artificial intelligent interactive chat bot with the goal of improving medication adherence and patient outcomes in three integrated healthcare delivery systems.

Methods The Theory of mHealth, the Expanded RE-AIM/PRISM, and the PRECIS-2 frameworks were used for program planning, implementation and evaluation, along with a focus on dissemination and cost considerations. During the planning phase, the Nudge study team developed and piloted a technology-based nudge message and chat bot of optimized interactive content libraries for a range of diverse patients. Inclusion criteria are very broad and include patients in one of three diverse health systems who take medications to treat hypertension, atrial fibrillation, coronary artery disease, diabetes, or hyperlipidemia. A target of approximately 10,000 participants will be randomized to one of 4 study arms: usual care (no intervention), generic nudge (text reminder), optimized nudge, and optimized nudge plus interactive AI chat bot. The PRECIS-2 tool indicated that the study protocol is very pragmatic, although there is variability across PRECIS-2 dimensions.

Discussion The primary effectiveness outcome is medication adherence defined by the proportion of days covered (PDC) using pharmacy refill data. Implementation outcomes are assessed using the RE-AIM framework, with particular focus on reach, consistency of implementation, adaptations, cost, and maintenance/sustainability. The project has limitations including limited power to detect some subgroup effects, medication complications (bleeding) and longer-term outcomes (myocardial infarction).  Strengths of the study include the diverse health care systems, a feasible and generalizable intervention, transparent reporting using established pragmatic research and implementation science frameworks, strong stakeholder engagement and planning for dissemination and sustainment.

Trial registration The study was funded by the NIH grant number is 4UH3HL144163-02 issued 4/5/19. The Clinicaltrial.gov identifier number is NCT03973931 which was registered on 6/4/2019. URL: https://clinicaltrials.gov/ct2/show/NCT03973931

Contributions To The Literature

• This protocol addresses implementation and effectiveness of a unique text and phone based medication adherence program across different cardiovascular conditions and types of health care settings

• This project develops technology-based mHealth ‘nudge’ messages using longitudinal engagement of multiple stakeholders including patients, physicians, pharmacists and administrators

• Evaluation is conceptually based using the updated Expanded RE-AIM/PRISM framework

• We illustrate program and design planning using PRECIS-2 and designing for dissemination and sustainment strategies to produce a pragmatic trial

• The balanced inclusion of intervention fidelity assessment, study of adaptations, and detailed cost assessment addresses important issues for implementation science.

Background

Up to 50% of patients do not take their cardiovascular (CV) medications as prescribed, [1-6] resulting in increased morbidity, mortality, and healthcare costs [4,7,8].  Interventions to improve adherence, such as patient education, reminders, pharmacist support, and financial incentives, have produced mixed results—some demonstrating benefits, but many producing small to negative results [9-14]. Adherence interventions have been limited by 1) including adherent patients who may not need an intervention; 2) resource-intensive approaches involving pharmacists or behavioral health providers, often working outside their normal scope of work; and 3) lack of attention to evidence-based strategies to motivate behavior change [6].  

Brief behavioral interventions can influence decision-making and are impactful. Principles of behavioral economics have been incorporated into health interventions to “nudge” people to achieve improved health outcomes [15]. A behavioral nudge is a small change in framing choice that alters people’s behavior in a predictable way. A prior study testing financial incentives through elimination of copayments for cardiovascular medications in the year after acute myocardial infarction improved adherence by 4% to 6%; however, financial incentives are not generalizable and are unlikely to be sustainable [16,17]. Behavioral nudges such as commitments (e.g. asking patients for demonstrated commitment to change through a pledge), norms (using examples of others who take action), and salience (making information or recommendations resonant through use of stories) build on a well-evidenced body of behavioral science theory and have been shown to improve health behaviors such as smoking cessation and weight loss[15,18]. These have yet to be tested to improve medication adherence.

Mobile and digital technologies for health promotion and disease self-management [19-21] offer intriguing and yet untested opportunities to adapt behavioral ‘nudges’ using ubiquitous cell phone technology to facilitate medication adherence. As described below, the NUDGE study contains several elements to make it pragmatic, including applying the PRECIS-2 criteria and the Expanded CONSORT diagram [22,23]. These include diverse health systems and a large and diverse number of patients to evaluate generalizability. There are very few exclusion criteria and the intervention is largely automated. Measures include those important to health systems and patients and most are available using unobtrusive measures such as electronic health record data and tracking of study implementation activities.

The objectives of the multi-center NUDGE study are as follows over the course of 4 years:

1. Conduct a pragmatic patient-level randomized intervention across three health care systems (HCS) to improve adherence to chronic CV medications. The primary effectiveness outcome will be medication adherence defined by the proportion of days covered (PDC) using pharmacy refill data. Secondary effectiveness outcomes will include intermediate clinical measures (e.g., BP control), CV clinical events (e.g., hospitalizations), and healthcare utilization.
2. Evaluate the intervention and implementation using a mixed methods approach and applying the Expanded RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework. Key implementation outcomes will include reach, implementation consistency and adaptations, costs, and program maintenance. In addition, NUDGE assess the context and implementation processes to inform local tailoring and eventual expansion of the intervention within the three HCS more broadly and nationally.

Methods

Study Settings

The study will be conducted within three HCS: the VA Eastern Colorado Health Care System, the Denver Health and Hospital Authority, and the University of Colorado UCHealth. Patient, provider and health system stakeholders affiliated with each participating health system provided input on the development of the project and have committed to participate in all aspects of the study in the pragmatic clinical trial phase.

VA Eastern Colorado Health Care System

The VA Eastern Colorado Health Care System (VA) is comprised of the Rocky Mountain Regional (RMR) VA Medical Center (Aurora, Colorado) and its affiliated clinics. The health system is comprised of 12 community-based outpatient care clinics throughout the state. Our study will enroll patients from eastern Colorado. The VA Health Care System has a single electronic health record (i.e., Computerized Patient Record System [CPRS]) and a centralized corporate data warehouse (CDW) that houses clinical and pharmacy data. It is estimated that ~70% of VA patients obtain their medications through VA pharmacies. The rest get them through outside pharmacies.

Denver Health and Hospital Authority

Denver Health (DH) is an integrated health care system that serves as the primary health care safety net for the City and County of Denver and serves an estimated one in four Denver residents, or 208,000 people, per year. Our study will enroll patients from primary care clinics located throughout the Denver Metro region. The DH technology infrastructure includes an EPIC electronic health record (EHR) and a data warehouse that integrates patient clinical and administrative data, including pharmacy data. It is estimated that approximately two-thirds of DH patients obtain their medications through DH pharmacies.

UCHealth

UCHealth (formally known as University of Colorado Health) currently includes 11 hospitals across the state, hosting over three million annual outpatient visits and covering 19% of Coloradans plus referrals from surrounding states. Our study will enroll patients from primary care clinics located in the Denver Metro region. The system’s infrastructure includes a single instance of the EPIC EHR for all inpatient and outpatient activities. Patients currently get their medications at both UCHealth pharmacies and pharmacies outside of UCHealth. Our plan is to get medication refill data through Surescripts, which is a health information technology company that supports e-prescription and the electronic transmission of prescriptions between health care organizations and pharmacies. Surescripts provides information to UCHealth about medication filled outside of the health system.

All development, enrollment, allocation, and analytic features are outlined in Figure 1.

Patient Population

Identifying patients with clinical conditions and prescribed medication classes of interest

Designated HCS programmers identify eligible patients based on the presence at least 1 of the cardiovascular conditions listed in Table 1 and with a filled prescription for at least one of the classes of medications to treat these cardiovascular conditions within 100 days prior to study eligibility. Participants must have had a primary care clinic visit at one of the clinics at UCHealth or Denver Health, and within two years (VA; due to differences in the electronic health record (EHR) systems) of the time of the data pull.

International Classification of Diseases, Ninth and Tenth Revision (ICD 9 and 10) codes identifying the clinical conditions of interest were compiled by each participating HCS. National Drug Codes (NDC) or system specific medication codes were used to identify the medication classes of interest. On a quarterly basis, we identify new patients who have met eligibility criteria based on the clinical condition and filling one of the classes of medication of interest.

Exclusion criteria were purposefully kept minimal to maximize generalizability including: patients who a) do not have a mailing address listed in EHR; 2) do not have a landline or cellphone listed in EHR; 3) are currently pregnant if denoted in the EHR at the time of the data pull; or 4) have a mailing address outside of the state of Colorado; 5) primarily communicate in  a language other than English or Spanish; and 6) have a referral to hospice or palliative care.

All eligible patients identified above, are sent an opt-out consent packet. The packet contains an introductory letter with information about the study, an opt-out form, an opt-out survey, and a self-addressed, stamped envelope. We will send materials to patients in their preferred language if that preference is denoted in the EHR (i.e., English or Spanish). All materials will be sent on letterhead and branding appropriate and specific to each HCS. The letter will be signed by either the primary care provider for the patient or the site principal investigator. We will provide patients with 4 weeks to return the opt-out postcard from the date that the introductory study packet was sent out. If they did not return the opt-put postcard by this period of time, the patient is eligible for the study. The CONSORT diagram in Figure 1 shows our anticipated sample size, participation and attrition rates.

Identifying patients eligible for randomization

Patients meeting the above screening criteria are followed prospectively for pharmacy refill gaps using HCS-specific pharmacy data. Daily pharmacy data are obtained within each HCS with prescription information including a patient identifier, medication name, medication class, release date and days’ supply for each fill.  We determine a date that each medication class is due to be refilled (expected refill date) for each patient.  This expected refill date is calculated as the date of the last fill for a specified medication class plus its days’ supply. We adjust this expected refill date after considering factors such as medication supply on-hand using information up to 6-months prior to study eligibility, inpatient days assuming the medication is provided to the patient in this setting, and cancellation of a medication. 

Patients identified to have 7 or more consecutive days without filling a medication after this expected refill date (7-day gap) for a medication class listed in Table 1 at any time during the two-year monitoring period will be randomized. For patients who are prescribed multiple CV medications, eligibility for randomization will be triggered by the first 7-day gap for any medication. Once the study starts at a clinic, we randomize all patients who currently have a medication refill gap of 7 or more days. Once randomized, patients will remain in the same study arm for the entire study whether or not they have subsequent refill gaps. 

Randomizing patients who meet eligibility criteria

As part of the automated daily tracking of medication data, patients are identified as having an initial 7-day gap necessary for enrollment.  Once identified, we determine the randomization stratum based on the HCS and number of baseline medications and then randomize patients to 1 of 4 study arms: 1) usual care; 2) generic text; 3) Nudge text; and 4) Nudge text with AI chatbot.  Stratified randomization based on HCS as well as the number of baseline medication classes increases the likelihood of balance across groups for these key variables. 

Patients randomized to the usual care arm will not receive further study procedures. For patients randomized to study arms 2-4, we determine if the listed phone number is a landline or cellphone using a validation process available as part of our text messaging platform (Mobile Messenger; Upland Communications, Austin, TX). Patients with a landline phone number will receive interactive voice responses (IVR- processes outlined below) automated messages instead of text messages. All procedures were reviewed and approved by the Colorado Institutional Review Board on April 9, 2019. Patient recruitment began Nov. 11, 2019 and is estimated to be completed approximately July 30, 2022.

Pragmatic Trial Design

To assess the  level of pragmatism of the trial as designed, we engaged several investigators in a three step sequential evaluation guided by the Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2) [22,24], using methods previously designed by our group. In the first stage of evaluation, the co-principle Investigators, as well as two co-Investigators and a staff member with expertise in pragmatic trial design met to formally discuss the definitions of each PRECIS domain, taking care to specifically avoid discussion of how each would be scored in the Nudge trial. In the next phase, all four investigators independently scored the Nudge protocol along each dimension, returning their scorecards to the staff member without consulting with each other about scores. Finally, the investigators met once more to adjudicate final scores within each domain, discussing the merits and nuances of each until a final integer-score was reached (Figure 3). In general, Nudge is designed as a highly pragmatic trial, with consensus rankings across domains. The only domain scoring below a “4” was “Flexibility by Staff”, which was scored as a “3” owing to a moderately rigid conceptualization of how and when intervention is delivered to participating individuals.

Interventions and Implementation Strategies

Intervention development. The Nudge interventions and implementation strategies were developed using our Integrated Theory of mHealth framework and ‘designing and disseminating’ principles) [25-27]. This involved rapid, iterative, user centered design procedures and the resulting messages and delivery processes were then piloted and adapted to result in the procedures below.

Treatment Conditions

• Usual Care: This group will not receive an intervention. We have included a usual care group to demonstrate the impact of the text messaging interventions above and beyond usual care and changing context, given that many prior medication adherence interventions have demonstrated small to negligible effects.
• Generic nudge: A generic reminder text (see below) is delivered to patients to refill their medication at days 1, 3, 5, 7 and 10 after randomization.
• Optimized nudge: A behavioral nudge text (see below) is delivered to patients to remind them to refill their medications at days 1, 3, 5, 7 and 10 after randomization.
• Optimized nudge plus AI Chat Bot: A behavioral nudge text is delivered to patients to remind them to refill their medications at days 1 and 3 after randomization. If the patient has not filled their medication on days 5 and 9, in addition to receiving a behavioral nudge text, an AI algorithm delivers an interactive chat via a chat bot to assess barriers filling the medication.

The AI Chat bot assesses for common barriers to medication adherence 1) social determinants factors, 2) provider-patient/health care system factors; 3) condition-related factors; 4) therapy related factors and 5) patient-related factors using a script. Communication about all these barriers is pre-programmed to use as algorithms in the chat bot automated program. For each barrier, the AI Chat bot problem-solves with the patient and identifies commonly used successful approaches to overcome barriers, and asks patients to choose and enact one solution to improve medication adherence. The AI chat bot library includes algorithms to support specific strategies to circumvent the adherence barriers responsible for each instance of a medication refill gap. For example, patients are queried to determine if they have difficulty remembering what medications to take and when to take them; those that do are asked if using a medication diary, involving a caretaker, or setting an alarm on their phone would help. Patients are asked if they would like to try one of these strategies; for those that agree and identify a strategy, the AI chat program includes an algorithm to check in one week later to see how this strategy is going. This is illustrated in Figure 4.

Message transmission:

Patients with a cell phone number will receive text messages according to their assigned study arm via Mobile Messenger, an online platform specializing in text message transmission. Patients with a land line phone number (estimated at 5-10%) receive messages via (IVR) automated telephone messages. The IVR calls follow the same flow logic, message content, and frequency of calls as the text messages. They remain in the same study arm for the duration of the study and receive the same intervention for subsequent episodes of 7-day medication gaps.

Responding to text messages from patients

If a patient texts “stop” to unsubscribe, they are automatically and fully withdrawn from the study. Should a patient respond “done” to indicate they have already filled their prescription, we will hold sending patients any further text messages about refilling the medication in which they had a gap for 30 to 90 days based on the medication. If the patient has a refill gap for the same or another medications again, we will start delivering text messages within the same study arm to which they had been previously randomized. Patients may request Spanish messages at any time via text. We will start to deliver Spanish language texts following the request.

There will be text messages that do not fall into any of the categories above. A research assistant (RA) will monitor these responses and will triage the messages depending on the content of the messages, In our pilot study, some patients sent responses to the text messages that (a) requested additional information about the study and/or (b) requested more detail on the specific medication that required a refill, even though the text message they were responding to did not solicit this information. For these types of unsolicited messages, the RA will respond with a link to our study webpage where we will post information about the study, sponsors, participating institutions, and a contact number they can call for more information. This webpage will include a “Frequently Asked Questions” (FAQ) about Nudge link, and we will post responses to anticipated questions there, (e.g. “Does my provider know about this study?; How did you get my cell number?; What if I don’t want to participate?”; etc.) 

Patients may text unsolicited information about a side effect or adverse event related to their medications. In these cases, we will have the site study pharmacists call the patient to find out more about the issue. We will also have the pharmacist contact the patient’s provider to make them aware of the issue. Other responses, such as questions about the intervention, requesting information about their medication, or asking about cost or logistics of obtaining the medication. will by triaged and responded to by a RA, pharmacist, or physician, as deemed appropriate. We will catalogue the messages that we receive from patients and if there is a theme, we will develop a FAQ and place information on the study website.

If patients do not refill after the series of text messages.

In the cases where the patient still has not refilled their medication after 5 days of their reply, a study RA will first confirm that the medication refill has not been completed. If the medication has not been refilled after chart review, they will then contact the patient to see if there are having issues with refilling the medication and try to resolve any issues with the patient. The RA will follow the following script when contacting the patient.  As this is a pragmatic trial conducted in the context of usual care, concomitant patient care will be tracked, but will be permitted.

Study Evaluation

The study will be evaluated using the Practical, Robust Implementation and Sustainability Model (PRISM) and its component RE-AIM framework, and mixed methods assessments to identify key contextual factors and RE-AIM framework evaluation components of Reach, Effectiveness, Adoption, Implementation and Maintenance [28]. We will also develop tools and a sustainability plan to broadly disseminate and guide the intervention, if the intervention is found to be efficient. PRISM considers important implementation concepts from Diffusion of Innovations[29], the Chronic Care Model[30], the Model for Improvement [31], and the RE-AIM  framework [32] and highlights four components that influence implementation success: 1) organizational and participant characteristics; 2) intervention characteristics from the organizational (health care system and providers) and participants’ perspectives (i.e., patients); 3) implementation and sustainability infrastructure (e.g., training and support, job roles, audit and feedback systems); and 4) external environment (e.g. reimbursement policies, guidelines). These four elements will be assessed in a formative manner [33] and will be critical to understanding how to further disseminate the intervention if demonstrated to be efficient. We will incorporate the assessment of the four components that influence RE-AIM implementation outcomes into our evaluation and this is further discussed in the implementation evaluation below (see Table 2).

MEASURES AND ANALYSIS PLAN

REACH ANALYSIS PLAN:

We will use descriptive statistics to describe the following:

1. Number of eligible patients and their baseline characteristics.
2. Percent of patients who did not opt out. We have also included a questionnaire for patients who opt out on reasons for declining. We will describe the patients who opt-out and also return the questionnaire.
3. Percent of patients with 7-day gap
4. Representativeness of a) study participants compared to overall patients within each clinic and respective health system and b) of patients who opt-out vs. those who do not on age, sex, number of conditions, ethnicity and race.

EFFECTIVENESS analysis plan

The primary outcome is adherence to CV medications as measured by 12-month proportion of days covered (PDC), obtained using pharmacy records from each of the healthcare systems.  Secondary outcomes include clinical events (e.g., event times for stroke, MI, mortality), utilization of care (e.g., hospitalizations or clinic visits for CV-related reasons), and costs of the interventions and of medical care, and will be captured from the EHR at each HCS.  Subjects will be followed for 12 months after randomization to assess these primary and secondary outcomes. Subjects who have more than one year of follow-up (up to 3 years depending on when they are enrolled during years 2-3) will continue to be followed for secondary and longer-term maintenance outcomes.  All analyses will be based on the intent to treat principle, using all patients who were randomized.

The assessment of primary outcome PDC will be based on the number of outpatient days a patient has a medication available, relative to the number of days during which a patient was prescribed the medication and should have depleted their supply, excluding inpatient days and days following death.  The primary outcome will be average PDC, averaged across all medications the patient gapped on at baseline and is at risk of depleting on a day, and then averaged over all days when at risk of depleting at least one medication. 

Descriptive analyses will be used to describe the cohort and to check for balance across study arms within strata (clinics and number of other medications prescribed).  A simple ratio for primary outcome PDC will be calculated as number of days at risk that the patient had medication divided by number of days at risk, during the one-year period following treatment initiation.  These simple estimates of each patient’s PDC on each medication will be used for descriptive analyses.  Missing patient covariate data will be imputed using multiple chained equations and multiple imputation [34].

Formal analyses will be based on daily data, using a day-level Bernoulli model with logistic link for the number of days covered by medication, which will be 365 but excluding days not at risk of depleting as described above.  For a given medication, the model will include fixed effect terms for treatment arm, clinic, patient covariates, and a random subject effect for a subject’s tendency to have higher or lower PDC compared with other subjects.  Patient covariates will include MyHealthConnect use (a medication reminder system used by some UCHealth patients), patient demographics (age, gender, race, ethnicity), number of clinic visits in the prior year, number of other CV medications the patient is prescribed at baseline, and indicators for major baseline CV conditions (AF, CAD, diabetes, hyperlipidemia, hypertension). 

To estimate means and treatment effects of PDC on a linear (PDC difference) scale we will use methods of standardization (counterfactual calculations) [35] to estimate population average PDC for each treatment and each medication.  With this approach medication-specific models as above will be estimated using maximum likelihood and used to calculate the estimated probability a patient will have a given medication available on a given day, separately assuming they received each of the treatments, calculations which in some cases will be counterfactual.  These estimated probabilities will be used to calculate two types of population average estimates: a) Medication-specific PDC for each treatment, b) Average PDC for each treatment across all medications gapped on at baseline.  Each average is over all patients assuming they received each treatment, regardless of which treatment they actually received.  Treatment differences will be estimated from the relevant quantities.  Primary hypotheses involve pairwise comparisons between each of the four study arms, and will be conducted using a multistage gatekeeper approach to control for multiple comparisons [36]. Inference will be carried out using bootstrap methods.

Secondary clinical outcomes will be analyzed using similar approaches but based on appropriate models, e.g. Cox survival models for time to clinical event or rehospitalization.  Standardization methods again allow results to be expressed on interpretable scales such as risk difference [37]. Data will be analyzed using SAS (SAS Institute Inc., Cary, NC) and R.

We will use the methods described above and related methods to carry out additional analyses examining several types of moderation and mediation effects.  We will use interactions to examine heterogeneity of treatment effect (HTE) by drug class, health care system, and patient characteristics.  We will also examine mechanisms or mediators of treatment effect by considering treatment effects on direct responses to reminders, including time from reminder to refill, number of 7-day gaps, and measures of patient engagement, e.g. number of patient text responses to reminders (intensive text and chatbot arms only).

Statistical Power:

Required sample size was estimated for the primary outcome 12-month PDC using preliminary data from the VA, based on the following assumptions:  a) Two-sided level 0.05 tests, b) Power at least 80%; c) Difference between treatments in PDC of 10 percentage points; d) Bonferroni adjustment for the 6 pairwise comparisons among the 4 study arms, e) Analysis stratified by health care system, and f) Within-system and within-treatment residual standard deviation of 12 month PDC equal to 0.22 (mean 0.732), obtained by analysis of 2,859 veterans during the period 01/01/2017 – 12/31/2017 who were prescribed relevant medications. With these assumptions, and comparing any two treatments using a simplified analysis based on a linear model with the above residual standard deviation of PDC, we estimate that we will need N=119 subjects per treatment arm, total across the three health care systems, for a total of 476 subjects to be randomized across the three health care systems. To estimate available sample sizes we obtained data from each of the three HCS on the number of patients at 4 VA, 5 UCHealth, and 8 DH clinics on estimated numbers of patients with CVD conditions and prescribed CVD medications. Assuming that 75% of patients have a gap, another 15% of patients opt out of the study following randomization, and 10% of patients do not have usable outcome data, we expect to have outcome data for about 7,740 patients across the four study arms. Even with this conservative estimate, we expect to have ample subjects (nearly ten times as many as needed) to achieve the necessary power for the primary analysis of PDC.  Additional subjects will provide power for secondary analyses, and for analyses of secondary outcomes (see Figure 1). 

ADOPTION ANALYSIS PLAN

We will assess adoption in terms of the absolute number, proportion, and representativeness of the primary care clinics, physicians and pharmacists that begin implementation of the intervention compared to all primary care clinics within each respective health system. We will compare the structural characteristics (e.g., staffing levels, number of providers, and number of patients) of clinics that participate in the intervention compared to all primary care clinics within the health system. We will also assess the absolute number, proportion, and representativeness of a) providers and b) pharmacists at a given clinic who have patients randomized to the intervention and compared to those who do not have any participating patients. These findings will help guide a dissemination campaign. Finally, we will conduct brief interviews with those staff who decline to identify reasons why.

IMPLEMENTATION ANALYSIS PLAN

Implementation refers to the degree to which the intervention components and implementation strategies are implemented as intended,  adaptations made and costs of implementation (www.re-aim.org) [32]. Because the intervention is largely automated, we do not anticipate changes to intervention components. We still record any additions or modifications such as new or modified content to messages. It is more likely that some implementation strategies such as how providers are notified, exclusion criteria, how pharmacists interact with the nudge messages and how this project fits into the work flow at each HCS will occur. Adaptations will be assessed using a modified FRAME adaptation model [38]. We will employ rapid, mixed methods assessment methods to assess specifics of issues such as the timing, type, purpose, and source of adaptations [39].

In our implementation analysis plan, for fidelity we will address the issues outlined below. For most of these issues, we will use our study database and descriptive analyses (means, standard deviation, medians and ranges) as well as analyses of variance to compare different subgroups and to answer the questions below.

1. Among patients randomized to the intervention, how many text messages were delivered per patient, for which class of medication did the patient experience a gap, and did the patient have gaps on multiple medications during the course of the study
2. Proportion of patients enrolled in text message versus IVR
3. Among patients in arm #4 (optimized nudge plus AI chat bot), proportion where AI chat bot led to a patient response and the medication adherence barrier identified
4. Barriers and facilitators to implementation of the intervention (see key informant interviews below)
5. Differences among the three HCS and different pharmacists
6. Qualitative interviews focused on PRISM factors of : 1) organizational and participants characteristics; 2) intervention characteristics from the organizational (health care system and providers) and participants’ perspectives (i.e., patients); 3) implementation and sustainability infrastructure (training and support); and 4) external environment.

Economic Analysis Plan

This project includes two economic components. First, we will calculate the total cost of implementing each intervention to inform the resource use and investment required. Second, we will estimate the health care costs and cost offsets associated with the intervention arms to inform if there were reductions in healthcare utilization that resulted in overall cost savings.

To calculate the total cost of implementing each intervention arm we will use a direct measure micro-costing approach. We will measure activities associated with the intervention and assign costs to them. Costs will be calculated by multiplying the number of units consumed by the unit cost for each cost component. Total costs will be stratified by upfront and implementation costs. Upfront costs will include those costs necessary to initiate the intervention, but occur before implementation. These will include development of text and AI chat bot messages, translation of messages to Spanish, training of staff, etc. Implementation costs include those costs necessary to deliver the intervention. These may include the costs to send the text message to the randomized patients, the AI chat box services, etc.

Costs will be collected in a prospective fashion alongside the clinical trial and will include personnel and non-personnel costs. We have developed a log for involved personnel to record their time spent on intervention activities. The log captures resource use associated with the intervention and collects data on the activity that was done, who did the activity, the title of the person who did the activity, and how much time was spent on the activity. Non-personnel costs will be tracked through receipts and invoices paid. Personnel and non-personnel costs will be summed to generate the total cost of each intervention. All unit cost data will be adjusted to the same year US dollar through inflation and discounting. Costs will be stratified by cost type (upfront versus implementation), intervention arm, and HCS. The incremental intervention costs will be calculated by comparing each active intervention cost to the usual care cost. Pairwise comparisons between active interventions will also be calculated.

To estimate the health care costs and cost offsets associated with each intervention, EHR data from each HCS will be obtained on the healthcare utilization of patients in each study arm for a minimum of 12 months following implementation [35]. Using the same approach successfully employed in prior studies, we will extend the implications of this work by estimating healthcare costs from these utilization data. Once cost data are estimated from the utilization data using DRGs, RVUs, and AWP, they will be analyzed using the same models described for utilization and other study health outcomes with factors for study arm and health system. Generalized linear models (GLM) will be used. The primary dependent variable will be healthcare cost, in total and separated into inpatient, outpatient, and pharmacy cost buckets. The primary independent variable will be the intervention arm the patient was randomized to. Results will be stratified by healthcare system.

The usual care arm will be the referent group and healthcare cost estimates for each active intervention arm will be compared to the usual care arm. If healthcare costs in an active intervention arm are significantly less than the healthcare costs in the usual care arm, as evidenced by a negative beta on the intervention arm coefficient, the active intervention will be associated with cost savings. Analysis of component 2 will be limited by the fact that utilization data will only be specific to healthcare utilization that occurred at each HCS. Therefore, we will not be able to examine the association of each intervention on total healthcare cost, but instead will be able to examine the association of each intervention on each HCS healthcare cost.

MAINTENANCE ANALYSIS PLAN

Maintenance analyses will use most of the same variables and methods as for effectiveness (for the individual level) and adoption (at the setting and staff levels) but at a later time period ranging from 1-23 months after July of 2021 when enrollment ends.  The key questions to be answered are can the program be sustained over time across a) settings, b) patients and c) outcomes for patients. In addition, we will assess intent to continue or modify intervention following grant support, and if the intervention be extended to other settings patient populations with different contexts (see Dissemination Plan below).

Qualitative Analysis

Key Informant Interviews

 We will conduct key informant interviews with up to 3 providers and 2 pharmacists (6-9 across the 3 HCS) from each setting whose patients received the intervention to get their feedback about the intervention and the intervention effects on their patient’s medication taking behavior. If there is less than 90% participation among either providers or pharmacists, we will also conduct phone interviews of those who decline to participate about reasons they declined. Many providers will have received one or more messages from the study team informing them that their patient did not refill their medications and we will interview the providers on their perceptions of that process. We will also conduct key-informant interviews with HCS leaders (3-6 interviewees) in each setting who are responsible for institutional policies related to patient data-management, informatics and pharmacy. In these interactions, we will share findings from the research and gauge their reaction to the findings. With any indication of positive and efficient outcomes, we will ask participants to describe their likelihood to maintain the system within their setting, and to discuss any barriers to maintenance and specific actions needed to overcome these barriers or adapt the Nudge program in some way.

Assessment of patient perspectives

In year 4, after the intervention and follow-up period has ended, we will survey patients via text messaging using a previously developed text messaging survey (Figure 5). In a random sample of 80 patients who respond to the survey, we will also conduct brief telephone interviews to get more in-depth feedback on the intervention. The sample will be stratified evenly across patients who received one of the three text messages. We have conducted similar interviews with patients following adherence interventions. These interviews will help inform further refinement of the interventions as we plan for broader dissemination (if demonstrated to be an efficient intervention) to more clinics and patients with other chronic conditions.

Trial Status

This is version 2 of the study protocol (as of 9/1/2020) registered with ClinicalTrials.org and any further modifications to the protocol will a) be shared with and approval obtained from the IRB; and b) communicated to the Data and Safety Monitoring Board. Patient recruitment began Nov. 11, 2019 and is estimated to be completed approximately July 30, 2022.

DISSEMINATION PLAN

There are two key aspects to our dissemination plan, both informed by the recent literature and our team’s work on designing for dissemination [29-31,40-43]. 1) We conduct all activities including planning and stakeholder engagement activities with a focus on both a) continued use after our intervention and evaluation activities have concluded, and b) eventual use in diverse settings, ‘designing for dissemination’ from the outset, and updating/refining our plans throughout the project. 2) We will conduct different dissemination product development activities and then implement communication vehicles for each of our target audiences.

We have two primary target audiences: 1) HCS that could potentially adopt our Nudge program intervention components and implementation strategies that prove most efficient; and 2) medication adherence and illness self-management research or quality improvement teams that could use, replicate and extend findings using our protocols and resources.

For HCS, we will develop ’implementation and adaptation guides’ (sometimes referred to as playbooks) to help settings potentially interested in adopting our Nudge intervention (or its key components and core functions) in their system. Based on our ongoing experience in our Triple Aim QUERI program [43] developing such guides for different health system change projects, we will develop interactive ‘living documents [44]. In years 3-4 we will then pilot implementation of these resources to allow health systems to decide if this program could fit their HCS at the present time, and if so how they can a) ensure that key program elements and functions are delivered consistently, and b) that necessary and appropriate adaptations are made to make the program viable in their settings. In year 4 and follow-up grants we will then fully implement these dissemination/adaptation guides

Details of this process of developing interactive ‘living adaptation guides’ are discussed below in the Designing for Dissemination section. We will partner with our organizational stakeholders to identify the best dissemination venues and methods to reach this target audience. We expect these venues to include already existing meetings such as national community health centers, managed care, VA regional and national meetings, and national professional organization conferences that our target audience already frequents. These guides will reflect adaptations made over the course of the project as well as interviews with program implementers toward the conclusion of their intervention period. Adaptations will be identified via both interviews and tracking forms during years 2 and 3 [33,40,45] but will likely include technologic adaptations owing to the differing nature of site pharmacy refill data, new or different EHR systems or features, considerations for clinical follow-up processes, altered language in text messages to reflect local culture, and varying procedures to inform patients that they are involved in the messaging program.

Development of HCS implementation and adaptation guide

We will develop web-based implementation and adaptation guides that will compile the evidence-based findings from our research to support dissemination and implementation of the Nudge program in new settings. We will identify the core functions of the intervention as well as adaptable components [33,40,45,46] during years 2-3. First, we will create a scalability guide, focused on scaling up the intervention from a modest number of clinics within each of the three participating health care systems to system-wide implementation. This toolkit will include detailed steps to 1) identify nonadherent patients; 2) create linkages to text messages and/or AI Chat bot through Twillio and/or Textit, and 3) steps to download implementation data that will allow for ongoing audit and feedback. This guide can also be used to scale out for other HCS wishing to a) decide if this program is right for them (with or without adaptation), b) implement a Nudge program, or c) apply our principles and procedures for other healthcare issues.

The adaptation portion of the guide will offer guidance on how to use our theoretical frameworks for message design and optimization to create and pilot test messages relevant for other conditions and/or other medications. It will include examples of text messages, generic or behavioral nudges, examples of Chat bot conversations with patients, and examples of EHR notes to clinicians informing them that their patient has not refilled their medication. Similar to the guides we have developed for our QUERI projects, the guide will also include materials and templates reflecting the content of each intervention component, action-oriented recommendations, Frequently Asked Questions, guidance for future adaptations, resources required, and tools that have been found to facilitate successful implementation of the intervention. The final adaptation guides will be employed during year 4 for hypertension medication adherence and then in a follow-up grant, tested with other conditions.

Dissemination to research teams

Our second dissemination goal will be to enhance the science of dissemination research and to disseminate to research teams by transparently reporting our protocol in clincicaltrials.gov and the journal Implementation Science. In later years, we will publish our implementation results and lessons learned using the PRISM and RE-AIM frameworks in journal articles and present our findings at targeted national meetings including the annual NIH Dissemination and Implementation in Health meeting, the Academy Health annual research meeting, the Society of Behavioral Medicine meeting; relevant professional society meetings; and national and regional VA and HCS research meetings.

In addition, we will offer seminars, demonstrations and workshops that train other research teams in how to successfully implement and evaluate a Nudge program in their setting. Finally, we will share these materials with the other sites in the NIH Collaboratory, working with them and the coordinating center to make all scientific and pragmatic information available in ways that can be tailored for use in other sites.

Discussion

Two crosscutting approaches will be employed across our project that provide both great potential and likely operational challenges. These involve designing for dissemination (D4D) principles [27,42] and related approaches to enhance sustainment. Our D4D efforts will consist of: 1) user-centered rapid and iterative design methods [47,48] which are employed in Dr. Bull’s laboratory; and 2) significant and ongoing multi-level stakeholder engagement.

User centered design thinking and rapid development procedures 

We employed persuasive communication strategies with demonstrated efficacy for engaging participants in the social media realm. Critical to our iterative, user centered design procedures were consideration of message design to maximize access, which required attention to user literacy and numeracy. All messages were kept at or below a 5th grade reading level [49]. To this end, we spent three months in developing our message libraries for each intervention arm.
To ensure adequate involvement of potential users, we purposively sampled 20 participants from each of the three participating HCS--VA, DH and UCHealth—with a balance of older/younger patients, men/women, those with one versus multiple chronic CV conditions and native Spanish/English speakers. We generated and tested both English and Spanish text-messages and framed chat bot communications through multiple N of 1 (i.e. within subject) assessments that conformed to evidence-based strategies for persuasive message design [50-53]. This approach offered a way to quickly respond and iterate new versions of messages until consensus across participants, including for perceived efficacy and general acceptability, was reached. In later project years, we will use similar methods to develop and pilot test additional implementation strategies as needed to maximize adherence.

Stakeholder engagement

Significant and ongoing consumer involvement facilitates methods for including the perspective of potential adopting settings and implementation staff, as well as patients receiving the targeted services. We developed a stakeholder group with representation from each of our three HCS and included: primary care clinicians, pharmacists/pharmacy staff, clinical operations directors, local IT staff, and patients being treated for at least one of the five CV diagnoses of interest. There are three dimensions along which we have and will continue to maximize stakeholder engagement. First is the level of engagement as our stakeholders were involved in the proposal development and will be in significant ongoing ways integral to the project. Stakeholders are treated as members of the research team and engaged in an ongoing, iterative manner, answering conceptual, ethical, and operational questions through their participation. Importantly, stakeholders will continue to be involved throughout the life of the Nudge project and will contribute to initial design, rapid prototyping and subsequent iterations of the intervention, review of emerging results and development of communication and dissemination materials. Finally, we engage multiple types of stakeholders- not just patients and high-ranking organizational officials. We include a) patients and their families or significant others; b) ‘implementers’ or staff such as care managers and pharmacists that directly interact with patients and support the Nudge interventions; and c) decision makers. Specific stakeholders involved will vary over time and include different types of representatives- early on leaders responsible for deciding upon and gaining support for the program; during the middle stages- IT staff and supervisors responsible for deciding upon IT priorities and integrating Nudge with other HCS electronic technologies, and toward the later stages- those responsible for sustained funding, training, implementing Nudge, and supervising relevant staff.

Strengths, Limitations and Future Directions

Limitations

This project has both strengths and limitations. Limitations include that the three HCS are all integrated care systems, with some degree of centralized pharmacy services. Our results may not apply to less integrated HCS. Two strategies that we used to make the project more pragmatic: having minimal exclusion criteria at multiple levels (physicians, pharmacists and patients) and having strong, multi-level stakeholder engagement may increase the variability of interventions, implementation strategies and results, making it more challenging to detect significant effects. Stakeholder engagement could, like our decision to allow adaptations to each HCS, result in the study being less efficient, enhance variability or reduce ‘fidelity’ to the intervention. This said, we think these were correct decisions since our goal was to design and evaluate a pragmatic, feasible, cost efficient intervention likely to appeal to and involve different HCS, and be more likely to be maintained after the study period.

Strengths and Future Directions

The strengths of the project, which to some extent mirror the limitations above are: working with different HCS and the heterogeneity of patient populations; stakeholder engagement to tailor strategies to local settings; allowing, assessing and transparently reporting adaptations; and the generally pragmatic design. We are employing multi-level conceptual models (Expanded RE-AIM/ PRISM), transparent reporting criteria and addressing key implementation science issues such as adaptations and sustainability. In addition, unique aspects of the project include large scale use of interactive nudge technologies, the focus on pharmacists rather than relying solely on physicians, the economic analyses, and explicit designing for dissemination and sustainability. The results of this investigation should provide a strong basis for follow-up studies including broader dissemination to different HCS, comparative effectiveness research on variants of the most efficient interventions and implementation strategies identified, applications to different chronic conditions, and evaluation of long-term sustainment.

Declarations

Acknowledgements

We gratefully acknowledge Starlynne Gornail and Geoffrey Harger for their assistance with study coordination.  

Funding

This work was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by cooperative agreement UG3HLD144163 from the National Heart, Lung, and Blood Institute. This work also received logistical and technical support from the NIH Collaboratory Coordinating Center through cooperative agreement U24AT009676. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Under this cooperative agreement, the NHLBI is involved in the study design, plans for analysis, and interpretation. They are not involved in data collection, management or analysis. They were not involved in writing this report and do not have ultimate authority.

Availability of data and materials

The datasets used in the current paper (e.g., Estimates for the CONSORT Figure and the PRECIS-2 individual ratings) are available from the senior author on reasonable request. Study results are not yet available as this is a protocol paper.

Ethics declarations

Ethics approval and consent to participate

All patients who are eligible are automatically enrolled unless they opt-out. Once a participant has been identified as eligible for enrollment, they are sent a letter in the U.S. mail describing the study, the eligibility criteria and the process for automatic enrollment. A copy of this letter is provided as an Appendix. At that time, they are given instructions about how they may opt out if they do not want to participate. There is a second opportunity for them to opt-out. If they are flagged from system pharmacy records as not having filled a prescription for management of their cardiovascular condition, they will be assigned at random to a study arm and may begin receiving messages via text. At that time they are notified that they can opt-out by texting “STOP” in response to any message.  

This study was reviewed and approved by the Colorado Multiple Institutional Review Board, protocol number 18-2779.

Data and Safety Monitoring Board

Our Data Safety Monitoring Board consists of three senior scientists, each with substantial clinical trials experience. It includes: a professor in psychological clinical science, a leading pragmatic trials researcher and a mathematical statistician who meet quarterly to discuss the project, and then episodically as issues come up.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests

Authors' contributions

Authorship eligibility guidelines followed those of the International Committee of Medical Journal Editors and professional writers were not used. MH, SB and RG conceptualized the article and reviewed and integrated contributions of other authors. All authors contributed sections to the original grant and protocol submitted to NIH. RG coordinated article preparation and took major responsibility for translating our grant protocol into a journal article. All authors were responsible for writing text sections in their area of expertise. CK was especially instrumental in preparing sections on the dissemination plan and PRECIS-2. All authors read and approved the final manuscript.

References

1. Jackevicius CA, Mamdani M, Tu JV. Adherence with Statin Therapy in Elderly Patients with and Without Acute Coronary Syndromes. JAMA. 2002;288(4):462–7.
2. Jackevicius CA, Li PV, Tu J. Prevalence, Predictors, and Outcomes of Primary Nonadherence After Acute Myocardial Infarction. Circulation. 2008;117(8):1028–36.
3. Ho PM, Spertus JA, Masoudi FA, Reid KJ, Peterson ED, Magid DJ, et al. Impact of Medication Therapy Discontinuation on Mortality After Myocardial Infarction. Archives of Internal Medicine. 2006;166(17):1842–7.
4. Rasmussen JN, Chong A, Alter DA. Relationship Between Adherence to Evidence-Based Pharmacotherapy and Long-term Mortality After Acute Myocardial Infarction. JAMA. 2007;297(2):177–86.
5. Spertus JA, Kettlelkamp R, Vance C, Decker C, Jones P, Rumsfeld JS, et al. Prevalence, Predictors, and Outcomes of Premature Discontinuation of Thienopyridine Therapy After Drug-Eluting Stent Placement: Results from the PREMIER Registry. Circulation. 2006;113(24):2803–9.
6. Brown MT, Bussell JK. Medication Adherence: WHO Cares? Mayo Clinic Proceedings. 2011;86(4):304–14.
7. Wei L, Wang J, Thompson P, Wong S, Struthers AD, Macdonald TM. Adherence to statin treatment and readmission of patients after myocardial infarction: a six year follow up study. Heart. 2002;88(3):229–22933. http://heart.bmj.com/content/88/3/229.full.pdf
8. Wei L, Flynn R, Murray GD, MacDonald TM. Use and adherence to beta‐blockers for secondary prevention of myocardial infarction: who is not getting the treatment? Pharmacoepidemiology and Drug Safety. 2004;13(11):761–6.
9. Pearson SA, Lu CY, Ross-Degnan D, Soumerai SB. Interventions designed to improve the quality and efficiency of medication use in managed care: A critical review of the literature – 2001–2007. BMC Health Services Research. 2008;8(1):75. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2323373/pdf
10. Volpp KG, Loewenstein G, Troxel AB, Doshi J, Price M, Laskin M, et al. A test of financial incentives to improve warfarin adherence. BMC Health Services Research. 2008;8(1):272–272.
11. Murray MD, Young J, Hoke S, Tu W, Weiner M, Morrow D, et al. Pharmacist intervention to improve medication adherence in heart failure: a randomized trial. Annals of internal medicine. 2007;146(10):714–25.
12. Lee JK, Grace KA, Taylor AJ. Effect of a Pharmacy Care Program on Medication Adherence and Persistence, Blood Pressure, and Low-Density Lipoprotein Cholesterol: A Randomized Controlled Trial. JAMA. 2006 ;296(21):2563–71.
13. Peterson PN, Campagna EJ, Maravi MA, Allen LF, Bull SP, Steiner JM, et al. Acculturation and Outcomes Among Patients with Heart Failure. Circulation: Heart Failure. 2012;5(2):160–6.
14. Jalal ZS, Smith F, Taylor D, Finlay K, Patel H, Antoniou S. Impact of pharmacy care upon adherence to cardiovascular medicines: a feasibility pilot controlled trial. European Journal of Hospital Pharmacy. 2016;23(5):250–6. http://ejhp.bmj.com/content/early/2016/02/02/ejhpharm-2015-000790.full.pdf
15. Hoyt GM. Nudge: Improving Decisions About Health, Wealth, and Happiness. International Review of Economics Education. 2009;8(1):158–9.
16. Hirsh BJ, Smilowitz NR, Rosenson RS, Fuster V, Sperling LS. Utilization of and Adherence to Guideline-Recommended Lipid-Lowering Therapy After Acute Coronary Syndrome. Journal of the American College of Cardiology. 2015;66(2):184–189.
17. Volpp KG, Loewenstein G, Troxel AB, Doshi J, Price M, Laskin M, et al. A test of financial incentives to improve warfarin adherence. BMC Health Services Research. 2008;8(1):272–272.
18. Cialdini RB, Demaine LJ, Sagarin BJ, Barrett DW, Rhoads K, Winter PL. Managing social norms for persuasive impact. Social Influence. 2006;1(1):3–15.
19. Car J, Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R. Mobile phone messaging reminders for attendance at healthcare appointments. The Cochrane database of systematic reviews. 2012;(7):CD007458.
20. De Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. The Cochrane database of systematic reviews. 2012;12(12):CD007459.
21. Vodopivecjamsek V, de Jongh T, Gurolurganci I, Atun R, Car J. Mobile phone messaging for preventive health care. Cochrane Consumers and Communication Group. 2012;12(12):CD007457.
22. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. British Medical Journal. 2015;350:h2147. http://www.bmj.com/content/350/bmj.h2147.full.pdf
23. Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT Figure: Increasing Transparency in Reporting on External Validity. American Journal of Preventive Medicine. 2018;55(3):422–30.
24. Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17(32):32.
25. Chambers DA. Sharpening our focus on designing for dissemination: Lessons from the SPRINT program and potential next steps for the field. Translational behavioral medicine. 2019;XX;XX-XX. doi: 10.1093/tbm/ibz102
26. Knoepke CE, Ingle MP, Matlock DD, Brownson RC, Glasgow RE. Dissemination and stakeholder engagement practices among dissemination & implementation scientists: Results from an online survey. PloS one. 2019;14(11): e0216971. doi:10.1371/journal. pone.0216971
27. Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA. Designing for dissemination among public health researchers: findings from a national survey in the United States. American journal of public health. 2013 Sep;103(9):1693-1699.
28. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. The joint commission journal on quality and patient safety. 2008;34(4):228-243.
29. Rogers, EM. Diffusion of Innovations 5th Edition. Free Press; 2003
30. Wagner E. The Chronic Care Model. 2008. http://www.improvingchroniccare.org/index.php?p=The_Chronic_Care_Model&s=2
31. Berwick DM. Disseminating innovations in healthcare. JAMA. 2003;289(15)1969-1975. doi: 10.1001/jama.289.15.1969
32. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Frontiers in Public Health. 2019;7:64. doi: 10.3389/fpubh.2019.00064
33. Rabin BA, Mccreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, Multimethod Assessment of Adaptations Across Four Diverse Health Systems Interventions. Frontiers in public health. 2018;6:102
34. Buuren SV, Groothuis-Oudshoorn K, Gerarda M. Mice: Multivariate Imputation by Chained Equations in R. Journal of statistical software. 2011;45(3):urn:issn:1548–7660.
35. Zou, GY. Assessment of risks by predicting counterfactuals. Statistics in Medicine. 2009; 28(30):3761–81.
36. Dmitrienko A, Tamhane AC, Wiens BL. General Multistage Gatekeeping Procedures. Biometrical Journal. 2008;50(5):667–77.
37. Sjölander A, Dahlqwist E, Zetterqvist J. A Note on the Noncollapsibility of Rate Differences and Rate Ratios. Epidemiology. 2016;27(3):356–9. doi:10.1097/EDE.0000000000000433.PMID: 26680297
38. Stirman SW, Baumann AA, Miller CJ. The Frame: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science. 2019;14(1):58-68.
39. Palinkas L, Zatzick D. Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) in pragmatic clinical trials of mental health services implementation: methods and applied case study. 2019;46(2):255-270. doi: 10.1007/s10488-018-0909-3
40. Chambers DA, Norton WE. The Adaptome: advancing the science of intervention adaptation. American Journal of Preventive Medicine. 2016;51(4):S124–S131. doi: 10.1016/j.amepre.2016.05.011
41. Palinkas, LA, Aarons, G, et al. Mixed Methods Designs in Implementation Research. Admin and Policy in Mental Health. 2011;38(1):44-53.
42. Cohen EL, Head KJ, McGladrey MJ, Hoover AG, Vaderpool RC, Bridger C, Carman A, Crosby RA, Darling E, Tucker-McLaughlin M, Winterbauer N. Designing for dissemination: Lessons in message design from “1-2-3 Pap”. Health Commun. 2015;30(2):196-207. doi: 10.1080/10410236.2014.974130.
43. US Department of Veteran Affairs. QUERI – Quality Enhancement Research initiative. 2018. https://www.queri.research.va.gov/programs/triple_aim.cfm.
44. National Institutes of Health. NIH Collaboratory Living Textbook of Pragmatic Clinical Trials.2020. http://www.rethinkingclinicaltrials.org/.
45. Jolles MP, Lengnick-Hall R, Mittman BS. Core functions and forms of complex health interventions: a patient-centered medical home illustration. 2019;34(6):1032-1038. doi: 10.1007/s11606-018-4818-7
46. Bull S, Ezeanochie N. From Foucault to Freire through Facebook: toward an integrated theory of mHealth. Health Ed & Beh. 2016;43(4):399-411.
47. Bull S, Devine S, Schmiege SJ, Pickard L, Campbell J, Shlay JC. Text Messaging, Teen Outreach Program, and Sexual Health Behavior: A Cluster Randomized Trial. Am J Public Health. 2016;106(S1):S117–24.
48. Thompson JS, Matlock DD, McIlvennan CK, Jenkins AR, Allen LA. Development of a decision aid for patients with advanced heart failure considering a destination therapy left ventricular assist device. JACC: Heart Fail. 2015;3(12):965-76.
49. Gaglio B, Glasgow RE, Bull SS. Do patient preferences for health information vary by health literacy or numeracy? A qualitative assessment. J Health Commun. 2012;17 Suppl 3:109–21.
50. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013;2(1):10.
51. Demeyin WA, Frost J, Ukoumunne OC, Briscoe S, Britten N. N of 1 trials and the optimal individualisation of drug treatments: a systematic review protocol. Syst Rev. 2017;6:90.
52. Curro FA, Robbins DA, Naftolin F, Grill AC, Vena D, Terracio L. Person-centric clinical trials: defining the N-of-1 clinical trial utilizing a practice-based translational network. Clin Investigation. 2015;5(2):145–59.
53. Agency for Healthcare Research and Quality. Design and Implementation of N-of-1 Trials: A User’s Guide - Research Report - Final | AHRQ Effective Health Care Program. 2017. https://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=1844

Tables

Table 1: Inclusion criteria, cardiovascular conditions, and medications classes (pg.7)

 Table 2: RE-AIM Outcomes (pg.13)