Trial Design
The trial design was a cluster-randomized, type 2 hybrid trial. Following an exploration phase in which HIV-CBOs were recruited, HIV-CBOs (and their staff) were randomized to one of two strategies: 1) the ATTC strategy; or 2) the ATTC + ISF strategy. Following randomization, the trial was deployed using a multiphase design that included three 6-month phases corresponding to the preparation, implementation, and sustainment phases of the EPIS framework (16). During the implementation phase, HIV-CBOs recruited and randomized client participants to one of two clinical intervention conditions: 1) UC, or 2) UC + MIBI. Institutional review board (IRB) approval and oversight of all research activities were provided by RTI International’s IRB.
Context
HIV-CBOs, located in 23 states and the District of Columbia within the United States, provided the context for the SAT2HIV Project.
Participants
HIV-CBO staff participants. To be eligible to participate, an HIV-CBO was required to: (a) serve a minimum of 100 individuals living with HIV per year; (b) have at least two staff willing and able to be prepared to implement a MIBI for SUDs; and (c) have at least one leadership staff (e.g., supervisor, manager, director) willing to help ensure MIBI staff were given sufficient time to participate. There were no exclusion criteria. Each collaborating HIV-CBO identified two staff to be prepared to implement the MIBI as part of the project’s implementation phase and to be recruited for participation in staff surveys. Each HIV-CBO also identified 1–3 leadership staff to be recruited for participation in staff surveys. Staff completed surveys prior to randomization (i.e., before the preparation phase), after the implementation phase (month 13), and after the sustainment phase (month 19), and received a $25 e-gift card per survey. For more details, see the study protocol paper (14).
HIV-CBO client participants. Client eligibility was assessed by HIV-CBO staff via the project’s standardized screener. Eligibility criteria included: (a) having been diagnosed with HIV; (b) being 18 + years of age; and (c) acknowledging use of at least one substance in the past 28 days with self-reported endorsement of two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-5) criteria (66) for SUD for that substance during the past 12 months. An exclusion criterion was not being able to speak English, which was due to the project’s research staff and MIBI proficiency raters being monolingual. Clients who met eligibility criteria were recruited for study participation by one of several trained HIV-CBO staff. However, it was one of the two MIBI staff from each HIV-CBO who was trained to administer the baseline assessment, follow-up locator form, and open the randomization envelope with client participants. Each HIV-CBO was provided compensation to cover staff time to complete these research-related activities. Clients randomized to the UC + MIBI condition received the MIBI at no cost. Clients received a $20 gift card for completing the baseline assessment, as well as $20 for completing a 4-week follow-up assessment administered by research staff blinded to all condition assignments.
Implementation Strategies
Complementing the comprehensive descriptions provided as part of our study protocol paper (14), Table 1 highlights: (a) the 10 discrete strategies in the ATTC strategy, (b) the 7 discrete strategies in the ISF strategy, and (c) the phase that each strategy was made available. For the ATTC strategy, the HIV-CBO’s two MIBI staff were given the opportunity to receive 12 months of MIBI training and technical assistance: (a) training (5-hour online didactics, 2-day in-person workshop), (b) performance feedback (standardized feedback on 1–3 MIBIs during the preparation phase, standardized feedback on all MIBIs during the implementation phase), and (c) consultation (1–2 individual consultation calls during the preparation phase, monthly 1-hour group consultation calls during the implementation phase). For the ISF strategy, the HIV-CBO’s MIBI staff and leadership staff were given the opportunity to additionally receive 18 months of external facilitation led by one of the project’s ISF facilitators (monthly virtual team meetings lasting 30–60 minutes, 1–2 in-person meetings lasting approximately 4 hours). To maximize the extent to which the ISF strategy was implemented with consistency and quality, the project’s lead developer of the ISF strategy (BG) trained each ISF facilitator, reviewed randomly selected ISF session recordings (each virtual ISF facilitation meeting was video recorded for quality assurance purposes), and regularly supervised the ISF facilitators (no less than monthly, usually weekly). For more details see the study protocol paper (14).
Table 1
Discrete Strategies Included Within the ATTC Strategy and the ISF Strategy
| Discrete Strategies | Preparation Phase | Implementation Phase | Sustainment Phase |
ATTC Strategy | Centralized technical assistance | ✓ | ✓ | |
| Develop educational materials | ✓ | | |
| Develop and organize quality monitoring system | ✓ | | |
| Develop tools for quality monitoring | ✓ | | |
| Distribute educational materials | ✓ | | |
| Conduct educational meetings | ✓ | | |
| Make training dynamic | ✓ | | |
| Audit and provide feedback | ✓ | ✓ | |
| Provide ongoing consultation | ✓ | ✓ | |
| Create a learning collaborative | | ✓ | |
ISF Strategy | External Facilitation | ✓ | ✓ | ✓ |
| Develop tools for quality improvement | ✓ | ✓ | ✓ |
| Organize implementation team meetings | ✓ | ✓ | ✓ |
| Identify and prepare champions | ✓ | ✓ | ✓ |
| Assess for readiness and identify barriers | ✓ | ✓ | ✓ |
| Conduct local consensus discussions | | ✓ | ✓ |
| Conduct cyclical small tests of change | | ✓ | ✓ |
Note: ATTC = Addiction Technology Transfer Center; ISF = Implementation & Sustainment Facilitation. See the study protocol paper(14) for detailed descriptions of each discrete strategy. |
Clinical Interventions
UC consisted of referral to formal addiction treatment, mutual-help services, or both. Clients randomized to UC + MIBI received the project’s 20-30-minute MIBI for SUD provided by one of the HIV-CBO’s prepared/trained MIBI staff. The MIBI was designed to motivate individuals living with HIV who have an SUD to change their substance use by (a) examining their reasons for change, (b) receiving feedback about common negative interactions of substance use and HIV-related health issues, (c) further developing importance or confidence to reduce or stop their primary substance use, and (d) making a plan for change. For more details, see the study protocol paper (67).
Outcome Measures
Organized by phase (i.e., preparation phase, implementation phase, and sustainment phase), Table 2 details the staff-level outcome measures (i.e., time-to-proficiency, implementation effectiveness, level-of-sustainment) and client-level outcome measures (i.e., days of primary substance use, number of substance-related problems, times engaging in risky behaviors, days of substance use treatment, days of medication non-adherence) collected.
Table 2
Measure Name (Phase) | Measure Description |
1. Time-to-proficiency (Preparation Phase) | A staff-level measure of the number of weeks between MIBI staff completing the in-person training and demonstrating proficiency in the project’s MIBI for SUDs. Developed for this project based on research by Saldana (89). Proficiency was determined by one of the project’s MI experts, who rated audio recordings of MIBI practice sessions using the Independent Tape Rater Scale (ITRS) (100). The ITRS is used to rate 10 MI-consistent items for adherence and competence along 7-point scales. MIBI proficiency was demonstrated when a single MIBI session had half the items rated at the mid-point or higher for both adherence and competence items. |
2. Implementation effectiveness (Implementation Phase) | A staff-level measure of the overall consistency and quality of MIBI implementation during the project’s implementation phase. Developed for this project based on research by Klein (21). P PPPFirst, the cumulative number of MIBIs implemented was summed and standardized for each MIBI staff (MIBI consistency). Second, the MIBI proficiency scores were summed and standardized for each MIBI staff (MIBI quality). Proficiency of each MIBI session was assessed by the project’s cadre of raters who were trained, calibrated, and supervised by one of the project’s MI experts. A quality score was calculated for each MIBI session by multiplying the corresponding adherence rating (ranged from 1–7) and competence rating (ranged from 1–7) and summing for all 10 of the MI-consistent items (ranged from 10–490). Finally, MIBI consistency and MIBI quality scores were summed and standardized. |
3. Intervention effectiveness (Implementation Phase) regarding: | |
3.1. Days of primary substance use | A client-level measure of the number of days client participants self-reported using their primary substance during the past 28 days and measured using a modified version of the Addiction Severity Index.(101) Client’s primary substance was identified at baseline by asking: “Of the substances that you have used in the past 4 weeks (not including tobacco), which one has been the biggest problem for you OR caused you the most problems?” At follow-up, client participants were reminded of the substance they had indicated was their primary substance. |
3.2. Number of substance-related problems | A client-level measure of the number of the 11 DSM-5(66) substance use disorder symptoms client participants self-reported recognition of regarding their primary substance and during the past 28 days. At follow-up, client participants were reminded of the substance they had indicated was their primary substance. |
3.3. Times engaging in risky behaviors | A client-level measure of the number of times client participants self-reported engaged in unprotected sex, injection drug use, or needle sharing during the past 28 days, which was developed for this project based on the Addiction Severity Index.(101) |
3.4. Days of substance use treatment | A client-level measure of the number of days client participants self-reported having attended residential treatment, outpatient treatment, or self-help group meetings during the past 28 days, which was developed for this project based on the Addiction Severity Index. |
3.5. Days of HIV medication non-adherence | A client-level measure of the number of days client participants self-reported having missed at least one dose of their HIV medications during the past 28 days, which was developed for this project based on the on the Addiction Severity Index.(101) |
4. Level-of-sustainment (Sustainment Phase) | A staff-level measure of the number of MIBIs delivered during the project’s 6-month sustainment phase and measured via MIBI staff self-report as part of the project’s sustainment phase survey. |
Note: BI = brief intervention; MIBI = motivational interviewing-based brief intervention. Intervention effectiveness was assessed in terms of the impact of the ISF strategy on each of the client outcome measures. |
Moderator Measures
Table 3 details the staff-level measures (i.e., implementation readiness, implementation climate, leadership engagement, tension-for-change, motivational interviewing experience, personal recovery status) hypothesized to moderate the impact of the ISF strategy on the staff-level outcomes.
Table 3
Staff-level Moderator Measures
Measure Name | Measure Description |
Motivational interviewing experience | A baseline measure of MIBI staff perception of their motivational interviewing experience. Measured by having MIBI staff select one five response options (i.e., none, beginner, intermediate, advanced, or expert). |
Personal recovery status | A baseline measure indicating if MIBI staff considered themselves to be in recovery from alcohol/drugs (0 = no; 1 = yes). |
Implementation readiness | A baseline measure representing MIBI staff perception regarding their organization’s readiness for implementing a brief intervention for SUDs. Measured using the average of 12 items developed by Shea (102). Each item (e.g., staff working on this project want to implement this change; staff working on this project are committed to implementing this change; staff working on this project will do whatever it takes to implement this change) was measured on a 5-point scale (1 = Disagree, 2 = Somewhat Disagree, 3 = Neither Agree nor Disagree, 4 = Somewhat Agree, 5 = Agree). Coefficient alpha = .95. |
Implementation climate | A baseline measure representing MIBI staff perception regarding the extent to which implementing brief intervention for SUDs is expected, supported, and rewarded within their organization. Measured using the average of the 6 items developed by Jacobs (103). Each item (e.g., staff working on this project are expected to use brief intervention for substance use with a certain number of clients; staff working on this project get the support they needed to use brief intervention for substance use with eligible clients; staff working on this project receive recognition for using brief intervention for substance use with eligible clients) was measured on a 5-point scale (1 = Disagree, 2 = Somewhat Disagree, 3 = Neither Agree nor Disagree, 4 = Somewhat Agree, 5 = Agree). Coefficient alpha = .91. |
Leadership engagement | A baseline measure representing MIBI staff perception regarding the extent to which their HIV-CBOs leadership is committed, involved, engaged, and accountable for implementing brief intervention for SUDs. Measured using 4 items developed for this project based on the leadership engagement construct described by Damschroder (15). Each item (i.e., to what extent was the leadership of this organization committed to the implementation of brief intervention for substance use; to what extent was the leadership of this organization involved in the implementation of brief intervention for substance use; to what extent was the leadership of this organization engaged in the implementation of brief intervention for substance use; to what extent was the leadership of this organization accountable for the implementation of brief intervention for substance use) was measured on 7-point scale (0 = not at all to 6 = highest extent possible). Coefficient alpha = .94. |
Tension-for-change | A baseline measure representing MIBI staff perception regarding the extent to which implementing a brief intervention for SUDS is important, needed, and desired. Measured using 3 items developed for this project based on the tension-for-change construct described by Damschroder (15). Each item (i.e., to what extent do staff working on this project believe implementation of brief intervention for substance use is important; to what extent do staff working on this project believe implementation of brief intervention for substance use is needed; to what extent do staff working on this project believe implementation of brief intervention for substance use is desired) was measured on 7-point scale (0 = not at all to 6 = highest extent possible). Coefficient alpha = .92. |
Note: MIBI = motivational interviewing-based brief intervention. |
The targeted sample size was estimated via power analyses with Optimal Design Software (68). For analyses of staff-level outcomes, it was estimated that 78 MIBI staff nested within 39 HIV-CBOs would provide 80% power to detect a statistically significant (p < .05) difference for effect sizes .67 or greater (14). For analyses of client-level outcomes, it was estimated that 1,872 clients, nested within 78 MIBI staff, nested within 39 HIV-CBOs would provide 80% power to detect a statistically significant difference for effect sizes .20 or greater (24).
Randomization Sequence Generation
For randomization of HIV-CBOs (i.e., the clusters), each HIV-CBO was allocated to one of two implementation strategy conditions (ATTC; ATTC + ISF) via an urn randomization process (69). Specifically, using survey data collected during the exploration phase from HIV-CBO staff, seven organizational-level factors (i.e., importance of substance use screening, importance of brief intervention for substance use, innovation-value fit, implementation strategy-value fit, implementation climate for MIBI, implementation readiness for MIBI, and implementation effectiveness for MIBI) were entered into an urn randomization program (70) that optimized the balance of the two implementation strategy conditions on these factors.
During the project’s implementation phase, HIV-CBOs randomized client participants to one of two intervention conditions (UC; UC + MIBI) via a blocked randomization sequence (blocking size of 6) generated via a blocked randomization program (71). Within each participating HIV-CBO, each MIBI staff had a lock box containing 36 sequentially numbered tamper-evident security envelopes containing a randomization slip indicating condition assignment. The randomization envelope was opened in front of the client participant. Staff updated a centralized recruitment tracking log monitored multiple times per week by research staff.
Blinding
It was not possible to blind HIV-CBOs and their staff to the assigned implementation strategy condition, but the project’s ATTC strategy staff and Independent Tape Rater Scale (ITRS) raters were blinded to implementation strategy condition assignment. Additionally, it was not possible to blind HIV-CBOs, their staff, or client participants to clinical intervention condition assignment, but the project’s research staff who conducted the follow-up assessments were blinded to all condition assignments.
Statistical Methods
Statistical analyses were conducted using an intention-to-treat approach. Staff-level outcomes were approximately normal and within-site variation was close to zero. A series of multilevel adjusted analyses was conducted, each of which controlled for project cohort and was weighted via a propensity score weight derived by regressing implementation strategy condition assignment on staff characteristics. The interaction between implementation strategy condition assignment and each hypothesized moderator was examined first, with main effects examined as appropriate.
Client-level outcomes had strong floor effects (e.g., 0 of 28 days) and/or strong ceiling effects (e.g., 28 of 28 days), which led to bimodal u-shaped, j-shaped, or inverted j-shaped distributions. Given these non-normal distributions, linear regression analyses were not appropriate. Rather, these types of distributions are appropriately addressed using zero-and-one inflated beta (ZOIB) regression after data are transformed to a proportion scale (i.e., 0 to 1). The ZOIB model is a mixture model with three parts: (a) a prediction of the probability of the ceiling effect vs. other values (i.e., the ceiling effect), (b) a prediction of the mean for values in between, but not including, the floor and ceiling effect (i.e., non-ceiling/non-floor effect), and (c) a prediction of the probability of the floor effect vs. other values (i.e., the floor effect). We fit three-level multilevel ZOIB models to account for nesting of client participants within MIBI staff and MIBI staff within HIV-CBOs using the R package developed by Liu (72). Each ZOIB model was adjusted for the baseline value of the respective outcome measure, client characteristics (i.e., age, White, male, heterosexual, transgender, married, high school or higher, alcohol as primary substance, engagement in HIV care), project cohort, randomization to ATTC + ISF condition, randomization to UC + MIBI condition, and the cross-level interaction between ATTC + ISF condition and UC + MIBI condition.