Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a parallel cluster-randomized type 3 hybrid trial

doi:10.21203/rs.3.rs-4614222/v1

Download PDF

Research Article

Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a parallel cluster-randomized type 3 hybrid trial

https://doi.org/10.21203/rs.3.rs-4614222/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: Substance use disorders (SUDs) have a serious adverse impact on the physical and mental well-being of people with HIV. Previously, using a 39-site dual-randomized type 2 hybrid trial design, findings from the Substance Abuse Treatment to HIV Care Project supported the Implementation and Sustainment Facilitation (ISF) strategy to improve implementation and effectiveness of a motivational interviewing brief intervention (MIBI) for SUD within HIV service settings across the United States (US). Building on this trial, this parallel cluster-randomized type 3 hybrid trial aimed to test the incremental effectiveness of a pay-for-performance (P4P), a form of the “alter incentive/allowance structures” strategy.

Methods: Twenty-six HIV service organizations, their staff participants (N=87), and their client participants (N=341) were cluster-randomized to one of two implementation conditions. The control condition included staff-focused training, feedback, and consultation (TFC) and team-focused implementation and sustainment (ISF). The experimental condition included TFC+ISF as well as P4P (TFC+ISF+P4P). P4P used financial incentives to reward MIBI implementation (US$10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality (US$10 per demonstration). In addition to these outcomes, past 4-week changes/reductions in client participant’s days of primary substance use and anxiety symptoms were examined.

Results: The addition of P4P had a large and significant effect on the number of MIBIs implemented (d=1.30, p<.05) and reduction in anxiety (d=-1.54), but there was no impact on days of substance use. P4P had large effects on MIBI quality (d=1.24) and MIBI implementation effectiveness (d=1.28), but these were not significant (p<.10).

Conclusions: P4P is a form of the “alter incentive/allowance structures” strategy. Its function is to reward the implementation of a clinical innovation. Rewarding implementation is consistent with the theory of implementation effectiveness, which suggests implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key antecedent of implementation effectiveness (i.e., the consistency and quality of implementation). We found that P4P had a significant, positive impact on MIBI implementation in HIV service settings, but client-level outcomes were mixed. Future research should examine the effectiveness of the P4P strategy to improve implementation and sustainment of other evidence-based innovations.

Trial registration: ClinicalTrials.gov: NCT04687917. Registered 12/18/2020.

implementation

pay-for-performance

facilitation

motivational interviewing

substance use disorders

HIV

hybrid trials

In the context of brief motivational intervention delivery for substance use disorders in HIV service settings, this research provides experimental evidence supporting pay-for-performance (P4P), a form of the “alter incentive/allowance structures” strategy to incentivize implementation of clinical innovations.
Supporting the theory of implementation effectiveness, this research found the addition of implementation rewards improved implementation beyond what was achieved when implementation was expected and supported via training, feedback, consultation, and facilitation only.
Reduction in anxiety is a potential clinical benefit of substance use disorder intervention implementation.
This research may provide a model for others interested in measuring implementation effectiveness.

The United States’ Ending the HIV Epidemic initiative, announced in early 2019, put forth an ambitious target of achieving 90% reduction in new HIV infections within 10 years [1]. Substance use disorders (SUDs) continue to pose a barrier to this goal [refs]. For example, in May of 2019, just a few months after the announcement of this initiative and before the COVID-19 pandemic began, Garner and colleagues (2022) [2] conducted a national survey to estimate the prevalence and negative impacts of five different SUDs among people with HIV (PWH) in the United States. Estimates varied by region of the United States, but overall, average prevalence rates were 42.3% for cannabis, 41.9% for alcohol, 34.6% for opioid, 32.2% for methamphetamine, and 28.1% for cocaine. In terms of the negative impacts that each SUD has on HIV care (e.g., being linked to HIV care, being retained in HIV care) and other important areas of life (e.g., having stable housing, being employed), results of the national survey found the most problematic was methamphetamine and the least problematic was cannabis. When the prevalence rate and negative impact estimates were multiplied to estimate the population-level negative impact, the top three most problematic use disorders were alcohol, methamphetamine, and opioids. Because COVID-19 has been reported to have increased the use of alcohol and other substances as well as the prevalence of anxiety [3–6], the rates of comorbid HIV, SUD and mental health problems may now be even greater.

HIV health and social service settings provide a critical yet under-utilized opportunity to address these adverse impacts [7–10]. According to a review by Hitch and colleagues (2019) [11], SUD screening within HIV settings is limited. In addition to highlighting the need for HIV settings to conduct systematic SUD screening using validated tools, they noted screening must be accompanied by strategies to facilitate delivery of brief interventions and referral to SUD treatment [11]. This recommendation is consistent with other calls to improve the integration of SUD services within HIV settings [7–10]. More recently, as part of its Overdose Prevention Strategy, the Department of Health and Human Services appropriated $43.7 million dollars of funding in 2022 to support provision of accessible, evidence-based, and culturally appropriate SUD services for people with HIV or at high risk for HIV [12], and further HIV-focused SUD care allocations can be anticipated. The current implementation research, which focused on testing the incremental effectiveness of a pay-for-performance (P4P) strategy to help improve the implementation of an empirically-supported motivational interviewing brief intervention (MIBI) for SUD within HIV settings [13], may help inform how such funding is utilized. The current research may also help inform the field of implementation science, which continues to need effective strategies for improving the implementation of evidence-based innovations within real-world practice settings.

The Substance Abuse Treatment to HIV Care (SAT2HIV) Project

In 2014, as part of its effort to address comorbid HIV and SUD, the National Institute on Drug Abuse funded the Substance Abuse Treatment to HIV Care (SAT2HIV) Project. Using a dual-randomized type 2 hybrid trial design, 39 HIV service organizations (HSOs) were randomized to one of two implementation conditions [14] and within each HSO, client participants were randomized to one of two intervention conditions [15]. The control implementation condition, based on research by Miller and colleagues (2004) [16], was a staff-focused strategy that provided 2 staff per HSO with training, performance feedback, and on-going consultation in implementing the MIBI with client participants. This is the strategy used by the Addiction Technology Transfer Center (ATTC), an intermediary organization funded by the Center for Substance Abuse Treatment, to prepare staff for implementing motivational interviewing. In addition to this strategy, a team-focused strategy called Implementation and Sustainment Facilitation (ISF) was provided to HSOs and their participating staff (2 MIBI staff and 1–3 leadership staff) randomized to the experimental condition. Grounded in the theory of implementation effectiveness [17–19], which posits implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key mechanism of change, the ISF strategy focused on optimizing the support dimension of implementation climate during monthly 30–60 minute meetings with the HSO’s MIBI staff and Leadership staff. These meetings were facilitated virtually by an ISF facilitator. Consistent with the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework [20, 21], the focus of ISF meetings varied by EPIS phase. All ISF meetings, however, were guided by the principles of motivational interviewing: engaging, focusing, evoking, and planning [22–24].

The control intervention condition was usual care only (i.e., passive referral [e.g., pamphlet] to a local treatment provider), and the experimental intervention condition consisted of usual care plus a single 15 to 30-minute MIBI. As detailed by Garner and colleagues (2020) (11), compared to the staff-focused training, feedback, and consultation (TFC) control strategy, the addition of the ISF Strategy (TFC + ISF) resulted in significantly greater levels of implementation; both a greater number of MIBIs being implemented and a greater level of MIBI quality (i.e., fidelity) scores. Moreover, compared to usual care only, usual care plus the MIBI significantly reduced daily use of their primary substance for clients within the TFC + ISF condition but not within the TFC only condition [13].

Rationale for Testing Pay-For-Performance

Notwithstanding SAT2HIV Project’s success supporting ISF, significant room for improvement in implementation effectiveness remained. The rationale for testing P4P was twofold. First, P4P directly addresses the reward dimension of implementation climate – a key mechanism of change according to the theory of implementation effectiveness [17–19]. Second, prior implementation research supported P4P as highly effective and cost-effective for improving implementation of an evidence-based SUD intervention for adolescents [25–27]. Notably, P4P is a form of Powell and colleagues’ (2015) [28] “alter incentive/allowance structures” strategy they defined as “work to incentivize the adoption and implementation of the clinical innovation.”

Rationale for Trial Design

The current study used a parallel cluster-randomized design (staff clustered within HSO [the unit of randomization]) to minimize the likelihood of contamination across the project’s two implementation conditions, and because the parallel cluster-randomized design may be preferable to alternatives, including the stepped-wedge trial design [29–32]. In addition to the use of a parallel cluster-randomized design, a type 3 hybrid trial was used because of the importance of examining both implementation outcomes and client outcomes [33–37], including anxiety symptoms that are strongly associated with SUDs among people with HIV [38] and which have potential for improvement following SUD intervention [39, 40].

Aims and Hypotheses

As illustrated in Fig. 1, the aims of this parallel 26-site cluster-randomized type 3 hybrid trial included testing the effectiveness of P4P for improving implementation outcomes as well as client outcomes. We hypothesized that relative to MIBI staff trained and supported using the TFC + ISF strategy (control condition), which had been the most effective and cost-effective strategy in the prior trial [13, 41], that MIBI staff trained, supported, and rewarded using the TFC + ISF + P4P strategy (experimental condition) would achieve significantly greater: (a) MIBI implementation consistency, (b) MIBI implementation quality, (c) MIBI implementation effectiveness (i.e., the consistency and quality of implementation), and client participants in the experimental condition would achieve significantly greater (d) reductions in days of primary substance use, and (e) reductions in anxiety symptom severity.

[FIGURE 1]

Trial Design

The design was a 26-site parallel cluster-randomized type 3 implementation-effectiveness hybrid trial [34, 37]. Guided by the EPIS framework [20] HSOs (and their staff) were recruited and randomized to one of two conditions (TFC + ISF or TFC + ISF + P4P) during the exploration phase. During the hybrid preparation-implementation phase the TFC + ISF strategy was available to HSOs (and their staff) in both conditions. The P4P strategy was available to MIBI staff cluster-randomized to the TFC + ISF + P4P condition from onset of the preparation-implementation phase. A MIBI staff transitioned to implementation by implementing a MIBI with a client participant. Thus, HSO’s were in a hybrid preparation-implementation phase until all MIBI staff had implemented the intervention with at least one client participant. The preparation-implementation phase was 12-months for the project’s first cohort and 6-months for the project’s second cohort. This unplanned difference was due to the project being stopped early when the principal investigator changed institutions. Approval and oversight of all research activities was provided by the Institutional Review Board of Advarra.

Context

The project’s participating HSOs (N = 26) were located in 13 states and the District of Columbia within the United States.

Participants

Staff participants. For an HSO to be eligible for the project, it had to receive funding from the Health Resources & Services Administrations’ Ryan White program, have two or more staff members (e.g., case manager, behavioral specialist) willing to be trained to implement a MIBI for SUD, and have at least one leadership staff (e.g., director, manager, supervisor) willing to help ensure the MIBI staff were provided sufficient time for participation. The participating HSOs identified individual staff to be invited to participate. Staff were required to be at least 18 years of age. There were not any exclusion criteria for HSO or staff participation. After staff provided informed consent to participate in the project, which was obtained electronically, they completed an online survey. Each participant received a US$25 e-gift card upon completion. The survey assessed background characteristics (e.g., age, biological sex, ethnicity, race, highest level of education), implementation climate using the six-item measure developed and described by Jacobs et al. (2014) [42], motivational interviewing knowledge and experience using a two-item index measure developed for this study and where staff self-rated their current baseline knowledge (0 = no knowledge at all, 1 = knowledgeable to a small extent, 2 = knowledgeable to a moderate extent, 3 = knowledgeable to a great extent) and experience (0 = none, 1 = beginner, 2 = intermediate, 3 = advanced, 4 = expert) using MI, setting-intervention fit using the six-item measure developed and described by Garner et al. (2022) [43], and organizational readiness for change using the 12-item measure developed and described by Shea et al. (2014) [44].

Client participants. To be eligible, clients had to be at least 18 years of age and HIV positive. There were no exclusion criteria for client participation. Client participants were recommended to receive a MIBI when they reported unhealthy alcohol use or endorsed two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria [45] for at least one substance during the past 12 months. Client participants received a US$10 e-gift card upon completion of the online SUD screener. MIBI-recommended client participants were asked to complete additional online assessments. These included a pre-MIBI assessment completed within the 24-hour period prior to receiving the MIBI and a follow-up assessment completed 4 weeks after receiving the MIBI. The MIBI was provided at no cost. Participants received e-gift cards upon completion of each assessment (US$20 for pre-MIBI assessment; US$10 for 4-week follow-up assessment). For the 4-week follow-up assessment, participants received a US$10 bonus for completion within 24 hours of the online link being emailed or texted to them or a US$5 bonus for completion within 25–48 hours of the online link being emailed or texted to them.

Strategies

As part of the SAT2HIV Project, the team-focused ISF Strategy was an effective adjunct strategy to the staff-focused TFC Strategy (ISF + TFC) [13]. As such, ISF + TFC was the control strategy and ISF + TFC + P4P as the experimental strategy in the SAT2HIV-II Project. Regarding temporality, ISF was the strategy HSOs and their MIBI staff received first. This was strategic and designed to help optimize MIBI staff’s early understanding and motivation for the TFC Strategy’s components. Importantly, for HSOs randomized to ISF + TFC + P4P, the ISF strategy was intended in part to accelerate MIBI staff’s understanding and motivation for achieving the P4P Strategy targets: MIBI consistency and MIBI Quality. Each strategy, described below and in the Table 1. The Strategies Timeline, Activities, and Resources (STAR) Table is a novel tool that includes a summary of information logged as part of the Strategies Timeline, Activities, and Resources (STAR) Log (see https://www.c-dias.org/implementation-guides-and-measures/).

Table 1

The Strategies Timeline, Activities, and Resources (STAR) Table
STRATEGY TIMELINE	STRATEGY ACTIVITIES
Temporality (i.e., the strategy sequence)	Specification (i.e., the name, activity target, and a general description)	Function (i.e., the activity purpose and targeted implementation or client outcomes)	Form (i.e., a precise description of the activity, and delivering actor as applicable)	STRATEGY Resources
1st Strategy Rollout #1: Jan 2021 – Dec 2021 Rollout #2: Nov 2021 – April 2022	ISF: A team-targeted strategy that uses principles of motivational interviewing to optimize participating HSO team’s success regarding implementing and sustaining the MIBI at high levels of consistency and quality	To engage and focus HSO’s implementation team on implementation effectiveness (the targeted outcome), and to help evoke team motivation and their plans for maintaining/ improving their HSO’s MIBI-related performance	Monthly virtual meetings, each 30–60 minutes, provided by a masters-level ISF-trained member of the project research team, and conducted separately for each participating HSO and their participating Leadership and MIBI staff	ISF Manual ISF Workbook
2nd Strategy Rollout #1: Jan 2021 – Feb 2021 Rollout #2: Nov 2021 – Dec 2021	Training: A staff-targeted strategy that helps teach HSO’s participating staff how to implement the MIBI	To help HSO’s participating staff acquire the level of knowledge and skill that is necessary for implementing the MIBI with client participants	The MIBI training manual, one 4-hour online asynchronous computer-delivered training, and three 4-hour virtual training sessions delivered by PhD-level MIBI trainer	MIBI Manual Tour of MI Training MIBI Virtual Training
3rd Strategy Rollout #1: Mar 2021 – Dec 2021 Rollout #2: Jan 2022 – April 2022	Feedback: A staff-targeted strategy that helps inform HSO’s trained staff about the quality of their MIBI implementation with client participants	To help HSO’s trained MIBI staff identify their strengths and weaknesses regarding their quality of MIBI implementation with client participants	A MIBI session quality feedback report, which is provided for each MIBI session and provided by the machine-learning based feedback system for motivational interviewing developed by Lyssn	MI rating platform MIBI Tracker
4th Strategy Rollout #1: Mar 2021 – Dec 2021 Rollout #2: Jan 2022 – April 2022	Consultation: A staff-targeted strategy that provides participating HSO staff with opportunities to receive additional MIBI training and to ask questions regarding how to improve their quality of MIBI implementation with client participants	To provide MIBI booster trainings that help enhance the knowledge and skill of MIBI staff, including any replacement MIBI staff who joined the project following a MIBI staff turnover and watch a recording of the project’s MIBI training workshop	Monthly virtual consultations, each 30–60 minutes, provided by a PhD-level MIBI trainer, available to all MIBI staff, with separate monthly consultations for each of the project’s two strategy conditions	MIBI Booster Trainings Consultation Guide
5th Strategy Rollout #1: Mar 2021 – Dec 2021 Rollout #2: Jan 2022 – April 2022	P4P: A staff-targeted strategy that provides HSO staff with the opportunity to receive a modest-sized financial incentive for their performance/ completion of pre-specified behaviors	To reward HSO staff for their successful performance/completion of one of the pre-specified behaviors	A monthly gift card reward to HSO staff for the total P4P incentive earned during the previous calendar month, with gift cards being distributed by the research project’s coordination team.	Gift Cards P4P Tracker
Note: ISF = Implementation & Sustainment Facilitation; P4P = Pay-for-performance; HSO = HIV Service Organization; MI = Motivational Interviewing;
MIBI = Motivational Interviewing Brief Intervention

Implementation and sustainment facilitation. The ISF is a team-focused strategy that seeks to optimize MIBI consistency and MIBI quality via elevating implementation climate (i.e., the extent to which MIBI staff perceive these dimensions of implementation effectiveness to be expected, supported, and/or rewarded) [13, 14]. It is team-focused in that it requires engagement of staff to deliver the intervention as well as leaders to help address implementation barriers (e.g., competing priorities, insufficient time). To help optimize staff understanding of the implementation initiative and motivation for the training provided as part of the preparation phase, it is recommended that the first ISF strategy meeting be completed prior to the initiation of the staff-focused training strategy. Consistent with other research that has used motivational interviewing as an implementation strategy [46, 47], ISF uses key principles of motivational interviewing (i.e., engaging, focusing, evoking, planning) as a guide.

In this project, each HSO engaged in a virtual monthly ISF strategy meeting that included three or more of the four motivational interviewing principles. These meetings lasted, on average, 30–60 minutes. An ISF workbook programmed in Microsoft Excel helped standardize the ISF strategy, document HSO staff attendance, document which motivational interviewing principles were employed, and document any meeting notes and/or action items. The ISF strategy was provided to each HSO by one of four master-level facilitators, two of which had provided ISF as part of the SAT2HIV Project. ISF meetings were recorded and a link to the meeting recording was sent to the HSO’s MIBI staff and leadership staff following each meeting. Weekly ISF team meetings supervised by a PhD-level ISF strategy facilitator were conducted for quality assurance purposes.

Training, feedback, and consultation. The TFC strategy, similar to the ATTC strategy in the SAT2HIV Project [13, 14], is a staff-focused strategy composed of three discrete strategies. As part of the current project, MIBI staff were provided access to the project’s MIBI manual, 4-hour online introduction to motivational interviewing training course, and 12-hour virtual training in the MIBI protocol. Although conducted by the same trainer as the SAT2HIV Project, who was and remains part of the Motivational Interviewing Network of Trainers, the 12-hour virtual training was a condensed version of the 2-day in-person training used in the prior trial. This approach was necessitated due to COVID-19 safety protocols. The training content provided as part of the project’s in-person training was condensed into three 4-hour virtual training sessions and was conducted over a four-week period. A one-week break between the second and third training sessions was provided to enable MIBI staff with time to practice the MIBI before the third training session. Free continuing education credits were provided for completion of both the online training and virtual training.

Feedback on the quality of MIBI implementation was provided via a machine-learning based feedback system for motivational interviewing [48]. For each MIBI session at least 10 minutes in length, a MIBI quality score was recorded. The range for each MIBI quality score was 0 to 12, with higher scores indicative of higher quality. MIBI staff were required to demonstrate at least average MIBI proficiency (a score of 4) via a practice role play session, which was based on the average MIBI proficiency provided as part of the MIBI effectiveness trial [13], hence sufficient for achieving intervention effectiveness. MIBI role play sessions were conducted with another MIBI staff at their HSO who played the role of the client using a standardized client scenario provided by the project.

In addition to training and feedback, MIBI staff were provided the opportunity to participate in group MIBI consultation meetings. Conducted separately for each condition, these meetings were provided monthly, were 30–60 minutes in length, and were conducted by the same trainer who conducted the virtual training sessions. In addition to providing didactic booster training regarding motivational interviewing and our project’s MIBI protocol, these group consultations provided MIBI staff with the opportunity to ask specific questions about how to improve the quality of MIBI implementation with client participants. Each consultation meeting was recorded and a link to the recording was emailed to the MIBI staff in the respective condition.

Pay-for-Performance. P4P is a staff-focused strategy that utilizes financial incentives and explicitly targets the reward dimension of implementation climate – the extent to which implementation is expected, supported, and rewarded [17–19]. For each MIBI session implemented, the MIBI staff received a US$10 incentive. Additionally, for each MIBI session at least 10 minutes in length and that received a 6 + quality rating via a machine-learning based feedback system for motivational interviewing [48], the MIBI staff received an additional US$10 incentive. This machine-learning based system enabled MIBI staff to receive immediate feedback on session quality, including whether their overall score met or exceeded the project’s pre-defined quality target. The project’s target was a score of 6, which was the 80th percentile score obtained by MIBI staff as part of the MIBI effectiveness trial described above [13].

For the project’s first cohort, P4P incentives that incentivized attendance at the group consultation meetings and ISF meetings were introduced after month 6. Approved by the project’s IRB, MIBI staff were asked to complete a new informed consent form to document their consent to receive a US$30 incentive per monthly group consultation meeting attended and a US$30 incentive per ISF meeting attended. Also approved by the IRB, leadership staff were asked to complete a new informed consent form that documented their consent to receive a US$30 incentive per ISF meeting attended. These additional P4P incentives were only for the HSOs and their participating staff that were randomized to the TFC + ISF + P4P condition. This revised P4P protocol was offered from the start for the project’s second cohort. During the first week of each month, research staff calculated the incentive amount earned by MIBI staff during the prior calendar month, sent a notification email to the staff, and had an e-gift card for the total incentive amount earned sent to the MIBI staff. There was not a limit on the amount MIBI staff were able to earn.

[Table 1]

Intervention

The project’s single session MIBI protocol has been shown to be effective in multiple settings [13, 49]. It was designed to help motivate an individual with comorbid HIV and SUD to change their primary substance use by: (a) examining their reasons for change, (b) receiving feedback about some common negative interactions between substance use and HIV-related physical and mental health issues, (c) increasing the importance or confidence to reduce or stop their primary substance use, and (d) making a plan for change [15].

Outcomes

Implementation outcomes. According to the theory of implementation effectiveness [17–19], implementation effectiveness is a multidimensional construct representing the consistency and quality of implementation by targeted users. Consistency is similar to measures like reach [50] and penetration [51] and quality is similar to measures like fidelity [51] and integrity [52]. MIBI consistency was operationalized as the total number of MIBIs a trained MIBI staff implemented during the project, with no minimum session length. MIBI quality was operationalized as the total quality score a trained MIBI staff demonstrated during the project. For a MIBI quality score to be generated a MIBI session had to be at least 10 minutes in length. Each MIBI at least 10 minutes in length received a quality score between 0 and 12, with higher scores indicative of higher quality. Following the standardization of each respective measure, these two measures were summed and standardized to create a measure of implementation effectiveness (i.e., the consistency and quality of implementation) [17–19].

Client outcomes. Consistent with its type 3 hybrid trial design, this project also examined the incremental impact of P4P on client outcomes 4-weeks post-MIBI. The two client outcomes of interest were change in days of primary substance use and change in anxiety symptom severity, both of which were based on client participant’s self-report. They were asked “out of the past 28 days (4 weeks), about how many days did you use [insert their primary substance].” The Generalized Anxiety Disorder 7-item (GAD-7) [53], with scores ranging from 0 to 21 (lowest to highest severity), was used to assess anxiety during the past two weeks. These client outcomes were aggregated for each MIBI staff. Thus, for each trained MIBI staff we computed two client outcome impact measures. One for their impact on days of primary substance use, and one for their impact on anxiety. A change score was computed for each possible client participant (i.e., those who completed the 4-week follow-up assessment) by subtracting their respective pre-MIBI assessment measure from their respective MIBI follow-up assessment measure. Then, for each respective change score measure, we aggregated to the level of the corresponding MIBI staff. Negative values are preferable as they represent greater reductions in the outcome. If a MIBI staff member was trained but implemented the MIBI with zero client participants, their client impact scores were zero (i.e., no impact, no return on investment).

Targeted sample size

The targeted sample size was 30 HSOs and 90 MIBI staff. Assuming an intraclass correlation of .06, power analysis indicated sufficient power to detect medium-sized effects.

Randomization sequence generation

For each project cohort, randomization of HSOs (the clusters) was completed by the project’s principal investigator, project coordinator, and statistician. The principal investigator and project coordinator independently matched pairs of HSOs based on information collected as part of an organizational background form (e.g., number of people with HIV served per year, number of staff employed, number of staff who have been trained in any type of motivational interviewing). They then met to compare rankings and reach consensus on a final list of match pairs. The final list of matched pairs was given to the statistician who randomized each matched pair to one of the two implementation conditions.

Blinding (Masking)

Client participants were blinded (masked) to implementation condition, as was the MIBI training staff. There was no blinding (masking) of implementation condition for the HSOs and their staff participants, or any other members of the study team.

Statistical methods

Statistical analyses were conducted using an intention-to-implement approach. All MIBI staff who received some component of the TFC strategy or ISF strategy were included in the analyses. All analyses were conducted using SAS version 9.4 [54]. An initial analytic step was to examine the partitioning of variance for each outcome measure. Mixed effects regression analyses were used for the impact of P4P on implementation outcomes and general linear regression analyses were used for the impact of P4P on client outcomes. Doubly robust estimation (i.e., combined propensity score weighting with outcome regression) was used to increase the models robustness to misspecification [55–58]. Factors used to create the staff propensity weight were age, gender, ethnicity, race, education, and experience, both with the organization and in their current position with the organization. All analyses controlled for project cohort. We examined the extent to which there were any significant interactions between condition assignment and the staff factors included in the model. If no significant interaction was found, we focused on the main effect of the implementation condition.

Participant flow and recruitment

Two cohorts of HSOs were recruited and randomized as part of the project. The first cohort, which cluster-randomized 12 HSOs, 44 MIBI staff, and 198 client participants, occurred from January 2021 through December 2021. The second cohort, which cluster-randomized 14 HSOs, 43 MIBI staff, and 143 client participants, occurred from November 2021 to April 2022. Figure 2 details the flow of the project’s 26 HSO (87% of targeted sample), 87 MIBI staff (97% of targeted sample), and 341 client participants. The targeted sample size of 30 HSOs and 90 MIBI staff was not achieved because the project’s principal investigator changed organizations and the trial was forced to stop early. A target number of client participants was not pre-specified because number of MIBIs implemented (MIBI consistency) was a primary outcome of interest.

[FIGURE 2]

Half of the 26 HSOs were randomized to TFC + ISF and the other half were randomized to TFC + ISF + P4P. In the TFC + ISF condition, one HSO withdrew from the project after randomization and before any of the project’s preparation phase strategies (ISF meetings, MIBI training) could be provided. The number of MIBI staff per condition was nearly the same, with 44 in TFC + ISF and 43 in TFC + ISF + P4P. The number of client participants in the TFC + ISF + P4P condition (n = 224) was 91% more than in the TFC + ISF condition (n = 117). The positive screening rate was 14 percentage points higher in TFC + ISF + P4P (80%) than in TFC + ISF (66%). There was little difference between conditions in terms of the number of client participants who withdrew from the study prior to completing the pre-MIBI assessment, with this rate being low for both conditions (10% for TFC + ISF and 8% for TFC + ISF + P4P).

In addition to the SUD screening assessment, client participants were asked to complete a pre-MIBI assessment within the 24-hour period prior to receiving the MIBI. If the pre-MIBI assessment and MIBI were not completed within 30 days of the SUD screen, the MIBI participant was documented as lost. The percentage of client participants lost between completion of the SUD screener and the pre-MIBI assessment was 18 percentage points lower in TFC + ISF + P4P (30%; 50 of 165) than in TFC + ISF (48%; 33 of 69). Rates of completion for the 4-week post-MIBI assessment were nearly identical between conditions: 66% (23 of 35) for the TFC + ISF condition and 67% (70 of 104) for the TFC + ISF + P4P condition. There were no significant differences between clients retained and clients lost to follow-up.

Participant characteristics

Table 2 presents the baseline characteristics for MIBI staff participants for the overall sample (N = 87) and each condition (43 for TFC + ISF; 44 for TFC + ISF + P4P). Most MIBI staff were female (63%), non-Hispanic (77%), non-White (63%), and did not have a graduate degree (72%). On average, MIBI staff were 40.0 (SD = 12.5) years of age, had 51.4 (SD = 67.6) months of experience with their HSO, and 37.3 months (SD = 53.1) of experience in their current position. Table 3 presents the characteristics of client participants who were recommended for and received the project’s MIBI overall (N = 139). See additional files for the characteristics of all client participants who consented to participate in the study and completed the SUD screener (N = 341) and the subset who were recommended for a MIBI (N = 257).

Table 2

MIBI Staff demographic characteristics
	Overall (N = 87)		TFC + ISF (N = 44)		TFC + ISF + P4P (N = 43)
	N	%	N	%	N	%
Age
18–24	4	4.6	2	4.6	2	4.7
25–34	35	40.2	23	52.3	12	27.9
35–44	17	19.5	7	15.9	10	23.3
45–54	13	14.9	5	11.4	8	18.6
55–64	15	17.2	7	15.9	8	18.6
65 and older	3	3.5	0	0.0	3	7.0
Female	58	66.7	31	70.4	27	62.8
Hispanic	27	31.0	17	38.6	10	23.3
Not Hispanic	60	69.0	27	61.4	33	76.7
White	39	44.8	23	52.3	16	37.2
Black	32	36.8	11	25.0	21	48.8
Asian	3	3.5	1	2.3	2	4.7
American Indian/Alaska Native	1	1.2	1	2.3	0	0.0
Native Hawaiian or Other Pacific Islander	1	1.2	1	2.3	0	0.0
Multi-Racial	1	1.2	1	2.3	0	0.0
No race selected, but Hispanic indicated	10	11.5	6	13.6	4	9.3
Graduate degree or higher	34	39.1	22	50.0	12	27.9
Experience at current organization
12 months or less	29	33.3	15	34.1	14	32.6
13–24 months	6	6.9	2	4.6	4	9.3
25–60 months	32	36.8	19	43.2	13	30.2
61–120 months	9	10.3	6	13.6	3	7.0
121 + months	11	12.6	2	4.6	9	20.9
Tenure at current position
12 months or less	32	37.2	16	37.2	16	37.2
13–24 months	13	15.1	6	14.0	7	16.3
25–60 months	31	36.1	19	44.2	12	27.9
61–120 months	4	4.7	2	4.7	2	4.7
121 + months	6	7.0	0	0.0	6	14.0
Baseline measures
Implementation Climate scale score (mean (SD))	87	2.2 (0.9)	44	2.1 (0.9)	43	2.4 (0.8)
MI knowledge and experience (mean (SD))	87	5.4 (1.6)	44	5.0 (1.3)	43	5.9 (1.8)
Setting Intervention Fit Scale Score (mean (SD))	87	17.3 (4.0)	44	16.5 (4.1)	43	18.2 (3.9)
Organizational Readiness for Change (mean (SD))	87	4.4 (0.6)	44	4.4 (0.6)	43	4.5 (0.6)

Note: TFC = Training, Feedback, and Consultation; ISF = Implementation & Sustainment Facilitation; MIBI = motivational interviewing-based brief intervention, P4P = Pay For Performance

Table 3

Client demographic characteristics (Received MIBI and Completed Follow-up Interview)
	Overall (N = 93 clients)		ISF + TFC (N = 23 clients)		ISF + TFC + P4P (N = 70 clients)
	N	%	N	%	N	%
Age
18–24	0	0.0	0	0.0	0	0.0
25–34	11	11.8	3	13.0	8	11.4
35–44	21	22.6	7	30.4	14	20.0
45–54	26	28.0	6	26.1	20	28.6
55–64	29	31.2	6	26.1	23	32.9
65 and older	6	6.5	1	4.4	5	7.1
Female	27	29.0	4	17.4	23	32.9
Hispanic	30	32.3	9	39.1	21	30.0
Not Hispanic	63	67.7	14	60.9	49	70.0
White	38	40.9	10	43.5	28	40.0
Black	43	46.2	8	34.8	35	50.0
American Indian / Alaska Native	6	6.5	2	8.7	4	5.7
Native Hawaiian or Other Pacific Islander	3	3.2	1	4.4	2	2.9
Multi-Racial	1	1.1	1	4.4	0	0.0
No race selected, but Hispanic indicated	2	2.2	1	4.4	1	1.4
Change in Outcomes Between Baseline and Follow-up
Days of Change in Primary Substance Use (mean (SD))	92	-3.32 (8.0)	22	-2.2 (7.4)	70	-3.7 (8.2)
Change in Anxiety GAD7 scale score (mean (SD))	92	-1.16 (5.1)	92	-0.26 (5.2)	92	-1.5 (5.0)

Note: TFC = Training, Feedback, and Consultation; ISF = Implementation & Sustainment Facilitation;
MIBI = motivational interviewing-based brief intervention, P4P = Pay-For-Performance

[Tables 2 & 3]

Strategy dosage

Table 4 presents information on strategy dosage. In the TFC + ISF strategy condition, the average dose was 12.86 hours (SD = 6.10) of training, 0.68 (SD = 1.61) times receiving feedback, 1.68 times (SD = 1.78) receiving consultation, and 3.52 (SD = 2.00) hours of ISF. Within the TFC + ISF + P4P strategy condition, the average dose was 13.12 hours (SD = 5.74) of training, 2.21 (SD = 1.61) times receiving feedback, 2.72 times (SD = 1.78) receiving consultation, and 3.90 (SD = 2.00) hours of ISF. There were no statistically significant differences between conditions.

Table 4

Strategy dosage for MIBI staff in the two conditions for each project rollout.
	Overall (N = 44 staff)		Control (N = 21 staff)		Experimental (N = 23 staff)
Rollout #1 (N = 12 HSOs): Jan 2021 – Dec 2021	Mean	SD	Mean	SD	Mean	SD
ISF (minutes)	246.49	23.19	225.76	137.41	267.22	170.20
Training (hours)	13.27	5.89	14.10	4.88	12.52	6.60
Feedback (times)	1.68	3.08	0.71	2.00	2.57	3.59
Consultation (minutes)	136.36	159.96	100.00	111.61	169.57	187.79
P4P earned (dollars)	N/A	N/A	N/A	N/A	26.52	37.60

	Overall (N = 43 staff)		Control (N = 23 staff)		Experimental (N = 20 staff)
Rollout #2 (N = 14 HSOs) Nov 2021 – April 2022	Mean	SD	Mean	SD	Mean	SD
ISF (minutes)	196.45	14.71	197.70	103.78	195.20	124.59
Training (hours)	12.70	5.94	11.74	6.83	13.80	4.47
Feedback (times)	1.19	2.46	0.65	1.13	1.80	3.30
Consultation (minutes)	126.98	133.36	101.74	102.31	156.00	156.92
P4P earned (dollars)	N/A	N/A	N/A	N/A	31.50	58.16

Note: MIBI = motivational interviewing-based brief intervention, ISF = Implementation & Sustainment Facilitation;
TFC = Training, Feedback, and Consultation; P4P = Pay-For=Performance. There were no statistically significant
differences between conditions.

[Table 4]

Outcomes

Analyses focused on testing the incremental impact of P4P (TFC + ISF vs. TFC + ISF + P4P) on implementation outcomes (Table 5) and client outcomes (Table 6). None of the staff background characteristics were found to significantly moderate the condition assignment to outcome relationship. This was true for all five outcomes. The addition of P4P had a large effect on all three implementation outcomes but was only statistically significant (p < .05) for MIBI consistency, β = 1.30, 95% confidence interval (CI) = [0.002, 2.6], p = .0497). For MIBI quality and MIBI implementation effectiveness, the results were β = 1.24, 95% CI = [-0.18, 2.7], (p = .08) and β = 1.28, 95% CI = [-0.08, 2.6], (p = .06), respectively. Total P4P costs were $870 for MIBI Consistency (staff range $0 to $120) and $370 for MIBI Quality (staff range $0 to $80).

Table 5

Results of analyses of the impact of the P4P strategy on implementation outcomes
	Full Sample
	Estimate (95% CI)	SE	p value
MIBI Consistency
ISF + TFC + P4P	1.30 (0.002,2.6)	0.56	0.0497**
Cohort 1	0.57 (-0.67,1.80)	0.54	0.32
Age	-0.002 (-0.04,0.04)	0.02	0.91
Female	-0.44 (-1.18,0.31)	0.32	0.22
Hispanic	0.37 (-1.03,1.76)	0.60	0.56
White	0.52 (-0.44, 1.47)	0.41	0.25
Graduate degree or higher	0.32 (-0.81,1.45)	0.49	0.53
Experience at current position	0.004 (-0.009,0.02)	0.005	0.52
Experience at current organization	-0.004 (-0.02,0.006)	0.005	0.37
MIBI Quality
ISF + TFC + P4P	1.24 (-0.18,2.67)	0.62	0.08*
Cohort 1	0.36 (-1.00,1.71)	0.59	0.56
Age	-0.006 (-0.05,0.04)	0.02	0.75
Female	-0.54 (-1.27,0.20)	0.32	0.13
Hispanic	0.39 (-1.08,1.85)	0.64	0.18
White	0.60 (-0.35,1.56)	0.41	0.24
Graduate degree or higher	0.63 (-0.52,1.78)	0.50	0.50
Experience at current position	0.004 (-0.009, 0.02)	0.005	0.23
Experience at current organization	-0.006 (-0.02,0.005)	0.59	0.56
MIBI Implementation Effectiveness
ISF + TFC + P4P	1.28 (-0.08,2.64)	0.59	0.06*
Cohort 1	0.46 (-0.83,1.76)	0.56	0.43
Age	-0.004 (-0.05,0.04)	0.02	0.83
Female	-0.49(-1.23,0.25)	0.32	0.17
Hispanic	0.38 (-1.05,1.81)	0.62	0.55
White	0.56 (-0.39,1.51)	0.41	0.21
Graduate degree or higher	0.48 (-0.66,1.62)	0.49	0.36
Experience at current position	0.004 (-0.008,0.02)	0.005	0.50
Experience at current organization	-0.005 (-0.02,0.005)	0.004	0.29

Note: CI = Confidence Interval; MIBI = motivational interviewing-based brief intervention;
ISF = Implementation & Sustainment Facilitation, TFC = Training, Feedback, and Consultation;
P4P = Pay For Performance.
*p <.0.10, **p<0.05, ***p<0.01

Table 6

The incremental impact of the P4P strategy on change in client-level outcomes
	Full Sample
	Estimate (95% CI)	SE	p value
Sum of PSU Days Changed
ISF + TFC + P4P	-0.34 (-1.63,0.95)	0.62	0.59
Cohort 1	-0.47 (-1.74,0.79)	0.61	0.45
Age	0.007 (-0.06,0.07)	0.03	0.82
Female	0.34 (-0.98,1.66)	0.64	0.60
Hispanic	1.00 (-0.66,2.67)	0.80	0.22
White	0.16 (-1.32,1.65)	0.72	0.82
Graduate degree or higher	0.93 (-0.57,2.43)	0.72	0.21
Experience at current position	0.01 (-0.008,0.03)	0.009	0.26
Experience at current organization	-0.01 (-0.03,0.003)	0.007	0.11
Sum of GAD7 Score Changed
ISF + TFC + P4P	-1.54 (-2.96,-0.11)	0.69	0.04**
Cohort 1	-0.36 (-1.76, 1.03)	0.67	0.60
Age	-0.009 (-0.08, 0.06)	0.04	0.80
Female	-1.66 (-3.12, -0.21)	0.70	0.03**
Hispanic	-2.63 (-4.47,-0.79)	0.89	0.007***
White	-0.50 (-2.13,1.14)	0.79	0.54
Graduate degree or higher	-0.52 (-2.18, 1.13)	0.80	0.52
Experience at current position	-0.008 (-0.03, 0.01)	0.01	0.41
Experience at current organization	0.008 (-0.009, 0.03)	0.008	0.31

Note: CI = Confidence Interval; PSU = Past Substance Use; GAD = Generalized Anxiety Disorder;
ISF = Implementation & Sustainment Facilitation, TFC = Training, Feedback, and
Consultation; P4P = Pay For Performance.
*p <.0.10, **p<0.05, ***p<0.01

In terms of client outcomes, overall reduction in days of primary substance was greater for MIBI clients assigned to the TFC + ISF + P4P condition, β = -0.34, 95% CI = [-1.63, 0.95], but was not statistically significant (p = .59). The addition of P4P had a significant impact on reduction in client anxiety symptoms, β = -1.54, 95% CI = [-2.96, -0.11], p = .04). However, there was a significantly greater overall reduction in level of client anxiety for those whose MIBI staff were female (β = -1.66, 95% CI = [-3.12, -0.21], p = .03) and/or Hispanic (β = -2.63, 95% CI = [-4.47, -0.79], p = .007).

[Tables 5 & 6]

There is growing interest in improving the integration of SUD services within HIV service settings [11, 9, 59–61]. Consistent with a systematic review highlighting the need for studies that “move beyond discrete training events and towards longer term coaching-type activities focused on implementation” [62] Garner and colleagues (2020) [13] found adding the ISF strategy (a team-focused coaching-type strategy) on top of the TFC strategy (a staff-focused training-type strategy) significantly improved both (a) the implementation of a MIBI for SUD within HSOs, and (b) the effectiveness of the MIBI in reducing substance use. Building on this research, the aim of the current 26-site parallel cluster-randomized type 3 hybrid trial was to test the incremental effectiveness of P4P (TFC + ISF vs. TFC + ISF + P4P) on both implementation outcomes and client outcomes.

In the current study, addition of P4P had a large and statistically significant impact on MIBI consistency (the number of MIBIs implemented), with the TFC + ISF + P4P condition implementing more than triple the number implemented within the TFC + ISF condition: 115 vs. 36. This large effect of P4P on implementation is consistent with prior research that found P4P to improve implementation of an adolescent substance use intervention [26].

The addition of the P4P strategy also had a large effect on the overall level of MIBI quality achieved by MIBI staff, but this effect was not statistically significant according to our pre-specific criteria of p < .05. However, we note that several methods-focused publications have recently argued against the use of this cutoff [63–65]. Rather, in contrast to the statistically significant effect found for MIBI consistency, the impact of P4P on MIBI quality was slightly smaller and short of achieving statistical significance due to insufficient statistical power because of having insufficient sample size. This was also the case for MIBI implementation. Our finding that P4P had a slightly less positive effect on quality than on quantity is in line with findings from a recent meta-analysis by Kim and colleagues (2022) [66]. More specifically, the meta-analysis by Kim et al. found that “the effect of incentives on performance was consistently and meaningfully larger for performance quantity than for quality,” which is a finding they highlighted as being consistent with cognitive evaluation theory [67–69] and its later incorporation into self-determination theory [70–73].

Regarding impact on client outcomes, MIBI staff in the TFC + ISF + P4P condition had slightly greater overall changes/reductions in days of primary substance use by clients compared to the TFC + ISF condition (d = − .34), but this was not at all close to being statistically significant (p = .59). Although also not statistically significant (p = .11), we did find that MIBI staff who had more experience with their organization had greater overall changes/reductions in days of primary substance use by clients. However, given that more experience working in the HSO was associated with poorer MIBI quality (though not statistically significant), we believe it may be possible that MIBI staff with greater experience working in their HSO were able to establish stronger therapeutic rapport/alliance with their clients; a relationship that much research has found to be one of the best predictors of improved client outcomes. Although beyond the scope of the current research, it may therefore be warranted for future research to rate MIBI sessions for therapeutic rapport/alliance and to examine its relationship with client changes/reductions in days of substance use.

We did find that client participants in the TFC + ISF + P4P condition had significantly greater changes/reductions in anxiety symptom severity (d = -1.54; p = .04) compared to those in the TFC + ISF condition. This finding is encouraging regarding the potential benefit of SUD intervention on anxiety outcomes. It must also be acknowledged that the more significant predictors of this outcome was if the MIBI staff was female (p = .03) or Hispanic (p = .007). There is research that has previously suggested a “female effect” [74] in better mental health outcomes, but we are not aware of research that has found similar effect for interventionists who identify as Hispanic. Future research exploring factors associated with improvement in anxiety may therefore be warranted.

limitations

The current research was not without limitations. First, both implementation conditions were affected, conducting the research during COVID-19 was not ideal and may have limited the extent to which the MIBI was able to be implemented. Second, the trial’s sample size and associated statistical power was lower than targeted, which as noted above in the methods section was due to the trial being stopped earlier than planned. Third, client outcomes were limited to two measures based on self-report. Fourth, only about two-thirds of follow-up assessments were completed, which was lower than the 70% rate targeted as a minimum follow-up rate, and therefore does adversely impact generalizability of client-level findings.

The consistency and quality of implementation (i.e., implementation effectiveness) is believed to be the greatest when it is expected, supported, and rewarded [17–19]. Building upon our prior research [13, 26, 27, 41], the current study findings support the effectiveness of using P4P to improve implementation of a MIBI for SUD within HIV service settings. Future research examining the effectiveness of P4P for improving the implementation of other clinical innovations that are both evidence-based and underutilized in real-world practice settings is warranted.

CI = Confidence Interval

DSM-V = Diagnostic and Statistical Manual of Mental Disorders

EPIS = Exploration, Preparation, Implementation, Sustainment

GAD-7 = Generalized Anxiety Disorder 7-item

HIV = Human Immunodeficiency Virus

HSO = HIV service organization

ISF = Implementation and Sustainment Facilitation

MIBI = Motivational Interviewing Brief Intervention

P4P = Pay for Performance

PWH = People with HIV

SAT2HIV = Substance Abuse Treatment to HIV Care

SD = Standard Deviation

SUD = Substance Use Disorder

TFC = Training, Feedback, and Consultation

US = United States

Ethics Approval and Consent to Participate

The current study was conducted under the auspices of the Advarra review board.

Consent for Publication

Not applicable.

Availability of Data and Material

Upon reasonable request, which should be made to the corresponding author, study data, resources, or other materials may be made available.

Competing Interests

None.

Funding

This work was supported by the National Institute on Drug Abuse (R01DA052294;
PI Garner).

Authors' Contributions

Conception and design of the work: BG, ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, JF. Acquisition, analysis or interpretation of data: ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, JF. Drafted or substantively revised the work: BG, ST, MB, KS, DS, CR, TD, JM, SP, RR, MZ, MR, JF.
In addition to having approved the submitted version, all authors have agreed to (a) be personally accountable for the author's own contributions and (b) ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Acknowledgments

The content is solely the responsibility of the authors and does not necessarily represent the official views of the government. The authors acknowledge that this project would not have been possible without both the participation and collaboration of the project’s 26 organizations and their staff, as well as the participation of the organization’s client participants. Additionally, this project would not have been possible without the assistance of Alyssa Toro and Liz Ball.

Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV Epidemic: A Plan for the United States. JAMA. 2019;321(9):844. doi:10.1001/jama.2019.1343
Garner BR, Gotham HJ, Knudsen HK, et al. The Prevalence and Negative Impacts of Substance Use Disorders among People with HIV in the United States: A Real-Time Delphi Survey of Key Stakeholders. Aids Behav. 2022;26(4):1183-1196. doi:10.1007/s10461-021-03473-9
Roberts A, Rogers J, Mason R, et al. Alcohol and other substance use during the COVID-19 pandemic: A systematic review. Drug and Alcohol Dependence. 2021;229:109150. doi:10.1016/j.drugalcdep.2021.109150
Schmidt RA, Genois R, Jin J, Vigo D, Rehm J, Rush B. The early impact of COVID-19 on the incidence, prevalence, and severity of alcohol use and other drugs: A systematic review. Drug and Alcohol Dependence. 2021;228:109065. doi:10.1016/j.drugalcdep.2021.109065
Taylor S, Paluszek MM, Rachor GS, McKay D, Asmundson GJG. Substance use and abuse, COVID-19-related distress, and disregard for social distancing: A network analysis. Addictive Behaviors. 2021;114:106754. doi:10.1016/j.addbeh.2020.106754
Satre DD, Iturralde E, Ghadiali M, et al. Treatment for Anxiety and Substance Use Disorders During the COVID-19 Pandemic: Challenges and Strategies. Journal of Addiction Medicine. 2020;14(6):e293-e296. doi:10.1097/ADM.0000000000000755
Sweeney S, Obure CD, Maier CB, Greener R, Dehne K, Vassall A. Costs and efficiency of integrating HIV/AIDS services with other health services: a systematic review of evidence and experience. Sex Transm Infect. 2012;88(2):85-99. doi:10.1136/sextrans-2011-050199
Savage CL, Sanchez M. Alcohol and Substance Use Disorder Screening, Brief Intervention, and Referral to Treatment Among People Living With HIV/AIDS. UJAN. 2016;27(3):214-217. doi:10.1097/JAN.0000000000000137
Haldane V, Cervero-Liceras F, Chuah FL, et al. Integrating HIV and substance use services: a systematic review: Haldane V et al. Journal of the International AIDS Society. 2017;20(1):21585. doi:10.7448/IAS.20.1.21585
The MASH Research Team, Scott-Sheldon LAJ, Carey KB, Johnson BT, Carey MP. Behavioral Interventions Targeting Alcohol Use Among People Living with HIV/AIDS: A Systematic Review and Meta-Analysis. AIDS Behav. 2017;21(S2):126-143. doi:10.1007/s10461-017-1886-3
Hitch AE, Gause NK, Brown JL. Substance Use Screening in HIV Care Settings: a Review and Critique of the Literature. Curr HIV/AIDS Rep. 2019;16(1):7-16. doi:10.1007/s11904-019-00434-9
Substance Abuse and Mental Services Administation. HHS Announces Nearly $44 Million to Strengthen Mental Health and Substance Use Services for Populations at Risk for or Living with HIV/AIDS.; 2022. Accessed March 29, 2023. https://www.samhsa.gov/newsroom/press-announcements/20220318/hhs-announces-44-million-mental-health-substance-use-population-hiv-aids#:~:text=Totaling%20%2443.7%20million%20dollars%2C%20the,HHS's%20new%20Overdose%20Prevention%20Strategy.
Garner BR, Gotham HJ, Chaple M, et al. The implementation and sustainment facilitation strategy improved implementation effectiveness and intervention effectiveness: Results from a cluster-randomized, type 2 hybrid trial. Implementation Research and Practice. 2020;1:263348952094807. doi:10.1177/2633489520948073
Garner BR, Zehner M, Roosa MR, et al. Testing the implementation and sustainment facilitation (ISF) strategy as an effective adjunct to the Addiction Technology Transfer Center (ATTC) strategy: study protocol for a cluster randomized trial. Addict Sci Clin Prac. 2017;12. doi:ARTN 32 10.1186/s13722-017-0096-7
Garner BR, Gotham HJ, Tueller SJ, et al. Testing the effectiveness of a motivational interviewing-based brief intervention for substance use as an adjunct to usual care in community-based AIDS service organizations: study protocol for a multisite randomized controlled trial. Addict Sci Clin Prac. 2017;12. doi:ARTN 31 10.1186/s13722-017-0095-8
Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A Randomized Trial of Methods to Help Clinicians Learn Motivational Interviewing. Journal of Consulting and Clinical Psychology. 2004;72(6):1050-1062. doi:10.1037/0022-006X.72.6.1050
Klein KJ, Sorra JS. The Challenge of Innovation Implementation. The Academy of Management Review. 1996;21(4):1055-1080. doi:10.2307/259164
Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational analysis. Journal of applied Psychology. 2001;86(5):811.
Helfrich CD, Weiner BJ, McKinney MM, Minasian L. Determinants of Implementation Effectiveness: Adapting a Framework for Complex Innovations. Med Care Res Rev. 2007;64(3):279-303. doi:10.1177/1077558707299887
Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Adm Policy Ment Health. 2011;38(1):4-23. doi:10.1007/s10488-010-0327-7
Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implementation Science. 2019;14(1):1. doi:10.1186/s13012-018-0842-6
Miller WR, Rose GS. Toward a theory of motivational interviewing. American psychologist. 2009;64(6):527.
Miller WR, Rollnick S. Motivational Interviewing: Helping People Change. Guilford press; 2012.
Wagner CC, Ingersoll KS. Motivational Interviewing in Groups. Guilford Press; 2012.
Garner BR, Godley SH, Dennis ML, Godley MD, Shepard DS. The Reinforcing Therapist Performance (RTP) experiment: Study protocol for a cluster randomized trial. Implement Sci. 2010;5. doi:Artn 5 10.1186/1748-5908-5-5
Garner BR, Godley SH, Dennis ML, Hunter BD, Bair CML, Godley MD. Using Pay for Performance to Improve Treatment Implementation for Adolescent Substance Use Disorders Results From a Cluster Randomized Trial. Arch Pediat Adol Med. 2012;166(10):938-944. doi:10.1001/archpediatrics.2012.802
Garner BR, Lwin AK, Strickler GK, Hunter BD, Shepard DS. Pay-for-performance as a cost-effective implementation strategy: results from a cluster randomized trial. Implement Sci. 2018;13. doi:ARTN 92 10.1186/s13012-018-0774-1
Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Sci. 2015;10(1):21. doi:10.1186/s13012-015-0209-1
Mdege ND, Man MS, Taylor CA, Torgerson DJ. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. Journal of clinical epidemiology. 2011;64(9):936-948.
Kotz D, Spigt M, Arts IC, Crutzen R, Viechtbauer W. Use of the stepped wedge design cannot be recommended: a critical appraisal and comparison with the classic cluster randomized controlled trial design. Journal of clinical epidemiology. 2012;65(12):1249-1252.
Kotz D, Spigt M, Arts IC, Crutzen R, Viechtbauer W. Researchers should convince policy makers to perform a classic cluster randomized controlled trial instead of a stepped wedge design when an intervention is rolled out. Journal of clinical epidemiology. 2012;65(12):1255-1256.
Kotz D, Spigt M, Arts IC, Crutzen R, Viechtbauer W. The stepped wedge design does not inherently have more power than a cluster randomized controlled trial. Journal of clinical epidemiology. 2013;66(9):1059-1060.
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challenges. Adm Policy Ment Health. 2009;36(1):24-34. doi:10.1007/s10488-008-0197-4
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact. Medical Care. 2012;50(3):217-226. doi:10.1097/MLR.0b013e3182408812
Foy R, Sales A, Wensing M, et al. Implementation science: a reappraisal of our journal mission and scope. Implementation Sci. 2015;10(1):51. doi:10.1186/s13012-015-0240-2
Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implementation Sci. 2020;15(1):84. doi:10.1186/s13012-020-01041-8
Curran GM, Landes SJ, McBain SA, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. FrontHealth Serv. 2022;2:1053496. doi:10.3389/frhs.2022.1053496
Mannes ZL, Hearn LE, Zhou Z, Janelle JW, Cook RL, Ennis N. The association between symptoms of generalized anxiety disorder and appointment adherence, overnight hospitalization, and emergency department/urgent care visits among adults living with HIV enrolled in care. J Behav Med. 2019;42(2):330-341. doi:10.1007/s10865-018-9988-6
Bahorik AL, Leibowitz A, Sterling SA, Travis A, Weisner C, Satre DD. Patterns of marijuana use among psychiatry patients with depression and its impact on recovery. Journal of Affective Disorders. 2017;213:168-171. doi:10.1016/j.jad.2017.02.016
Wolitzky-Taylor K. Integrated behavioral treatments for comorbid anxiety and substance use disorders: A model for understanding integrated treatment approaches and meta-analysis to evaluate their efficacy. Drug and Alcohol Dependence. 2023;253:110990. doi:10.1016/j.drugalcdep.2023.110990
Hinde JM, Garner BR, Watson CJ, Ramanan R, Ball EL, Tueller SJ. The implementation & sustainment facilitation (ISF) strategy: Cost and cost-effectiveness results from a 39-site cluster randomized trial integrating substance use services in community-based HIV service organizations. Implementation Research and Practice. 2022;3:263348952210892. doi:10.1177/26334895221089266
Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implementation Sci. 2014;9(1):46. doi:10.1186/1748-5908-9-46
Garner BR, Gotham HJ, Knudsen HK, et al. The Setting-Intervention Fit of Nine Evidence-Based Interventions for Substance Use Disorders Within HIV Service Organizations Across the United States: Results of a National Stakeholder-Engaged Real-Time Delphi Study. Jaids-J Acq Imm Def. 2022;90:S206-S214. doi:10.1097/Qai.0000000000002981
Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implementation Sci. 2014;9(1):7. doi:10.1186/1748-5908-9-7
American Psychiatric Association D, Association AP. Diagnostic and Statistical Manual of Mental Disorders: DSM-5. Vol 5. American psychiatric association Washington, DC; 2013.
Hettema JE, Ernst D, Williams JR, Miller KJ. Parallel Processes: Using Motivational Interviewing as an Implementation Coaching Strategy. J Behav Health Serv Res. 2014;41(3):324-336. doi:10.1007/s11414-013-9381-8
Kauth MR, Sullivan G, Blevins D, et al. Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implementation Science. 2010;5(1):1-11.
Imel ZE, Pace BT, Soma CS, et al. Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy. 2019;56(2):318.
Martino S, Ondersma SJ, Forray A, et al. A randomized controlled trial of screening and brief interventions for substance misuse in reproductive health. American journal of obstetrics and gynecology. 2018;218(3):322. e1-322. e12.
Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health. 2019;7:64. doi:10.3389/fpubh.2019.00064
Proctor E, Silmere H, Raghavan R, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65-76. doi:10.1007/s10488-010-0319-7
Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical psychology review. 1998;18(1):23-45.
Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Archives of internal medicine. 2006;166(10):1092-1097.
SAS version 9.4. (2023). SAS Institute Inc. Cary, NC, USA.
Robins JM, Rotnitzky A, Zhao LP. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association. 1994;89(427):846-866.
Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biometrics. 2005;61(4):962-973.
Funk MJ, Westreich D, Wiesen C, Stürmer T, Brookhart MA, Davidian M. Doubly Robust Estimation of Causal Effects. American Journal of Epidemiology. 2011;173(7):761-767. doi:10.1093/aje/kwq439
Vermeulen K, Vansteelandt S. Bias-reduced doubly robust estimation. Journal of the American Statistical Association. 2015;110(511):1024-1036.
Chichetto NE, Mannes ZL, Allen MK, Cook RL, Ennis N. HIV care provider perceptions and approaches to managing unhealthy alcohol use in primary HIV care settings: a qualitative study. Addict Sci Clin Pract. 2019;14(1):21. doi:10.1186/s13722-019-0150-8
Conway FN, Rountree MA, Jones KV. Serving the co-morbid mental health and substance use needs of people with HIV. Community Mental Health Journal. Published online 2021:1-12.
Goldhammer H, Marc LG, Chavis NS, et al. Interventions for Integrating Behavioral Health Services Into HIV Clinical Care: A Narrative Review. Open Forum Infectious Diseases. 2022;9(8):ofac365. doi:10.1093/ofid/ofac365
Savic M, Best D, Manning V, Lubman DI. Strategies to facilitate integrated care for people with alcohol and other drug problems: a systematic review. Subst Abuse Treat Prev Policy. 2017;12(1):19. doi:10.1186/s13011-017-0104-7
Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-307.
Kraemer HC. Is it time to ban the P value? JAMA psychiatry. 2019;76(12):1219-1220.
Wasserstein RL, Schirm AL, Lazar NA. Moving to a World beyond “P< 0.05.” Vol 73. Taylor & Francis; 2019:1-19.
Kim JH, Gerhart B, Fang M. Do Financial Incentives Help or Harm Performance in Interesting Tasks?
Deci EL, Cascio WF, Krusell J. Cognitive evaluation theory and some comments on the Calder and Staw critique. Published online 1975.
Deci EL, Koestner R, Ryan RM. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological bulletin. 1999;125(6):627.
Rummel A, Feinberg R. Cognitive evaluation theory: A meta-analytic review of the literature. Social Behavior and Personality: an international journal. 1988;16(2):147-164.
Deci EL, Ryan RM. The general causality orientations scale: Self-determination in personality. Journal of research in personality. 1985;19(2):109-134.
Deci EL, Ryan RM. Self-determination theory. Published online 2012.
Deci EL, Olafsen AH, Ryan RM. Self-determination theory in work organizations: The state of a science. Annual review of organizational psychology and organizational behavior. 2017;4:19-43.
Ryan RM, Deci EL. Self-determination theory. Basic psychological needs in motivation, development, and wellness. Published online 2017.
Bhati KS. Effect of client-therapist gender match on the therapeutic relationship: An exploratory analysis. Psychological reports. 2014;115(2):565-583.

Download PDF

Reviewers agreed at journal
20 Aug, 2024
Reviewers invited by journal
07 Jul, 2024
Editor assigned by journal
24 Jun, 2024
First submitted to journal
21 Jun, 2024

You are reading this latest preprint version

Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a parallel cluster-randomized type 3 hybrid trial

Status:

Version 1

Abstract

Figures

CONTRIBUTIONS TO THE LITERATURE

Background

The Substance Abuse Treatment to HIV Care (SAT2HIV) Project

Rationale for Testing Pay-For-Performance

Rationale for Trial Design

Aims and Hypotheses

Methods

Trial Design

Context

Participants

Strategies

[Table 1]

Intervention

Outcomes

Targeted sample size

Randomization sequence generation

Blinding (Masking)

Statistical methods

RESULTS

Participant flow and recruitment

[FIGURE 2]

Participant characteristics

[Tables 2 & 3]

Strategy dosage

[Table 4]

Outcomes

[Tables 5 & 6]

DISCUSSION

limitations

CONCLUSION

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1