Title: Keep It Up! 3.0: Study Protocol for a Type III Hybrid Implementation-Effectiveness Cluster-Randomized Trial

Few rigorously tested prevention interventions developed previous with high With nearly with going eHealth approaches to prevention may successfully bridge research and practice. Keep It Up! is an eHealth HIV Prevention program for young men who have sex with men. Previous research has demonstrated its effectiveness in reducing sexually transmitted infections and condomless anal sex and eciency in delivering HIV prevention education. Aim 1 is to compare two strategies for implementing KIU—implementation in community-based organizations and a centralized direct-to-consumer recruitment arm. Aim 2 is to examine adoption characteristics which explain variability in implementation success. Our exploratory aim will develop recommendations and materials for sustainment of KIU after completion of the


Abstract
Background Young men who have sex with men are disproportionately impacted by the HIV epidemic in the United States.
Few rigorously tested HIV prevention interventions have been developed for young men who have sex with men; previous interventions have primarily focused on in-person programming, with high variability in delity.
With nearly all young men who have sex with men going online daily, eHealth approaches to prevention may successfully bridge research and practice. Keep It Up! is an eHealth HIV Prevention program for young men who have sex with men. Previous research has demonstrated its effectiveness in reducing sexually transmitted infections and condomless anal sex and e ciency in delivering HIV prevention education. Aim 1 is to compare two strategies for implementing KIU-implementation in community-based organizations and a centralized direct-to-consumer recruitment arm. Aim 2 is to examine adoption characteristics which explain variability in implementation success. Our exploratory aim will develop recommendations and materials for sustainment of KIU after completion of the trial.

Methods
This is a Type III Hybrid Effectiveness-Implementation cluster randomized trial. Using estimates of young men who have sex with men per county in the United States, we identi ed 113 counties for our sample frame.
Using an iterative process, we selected 66 counties to randomize 2:1 to our two strategies in Aim 1. The RE-AIM model for implementation science will be used to drive our outcome measurements in reach, effectiveness, implementation variability, and cost. Outcome measures will be collected from communitybased organization staff participants, young men who have sex with men participants, and the technology provider. Our second aim will use mixed-methods research mapped onto the domains of the consolidated framework for implementation research.

Discussion
The trial has launched and is ongoing. This study is among the rst to use a cluster randomized trial design in HIV implementation science. In comparing the community-based organization and direct to consumer models for recruitment and ongoing participant engagement, we are examining two strategies which have shown effectiveness in delivering health and technology interventions in the past, but with little base knowledge on their comparative advantages and disadvantages in implementation. The results of the trial will further understanding of the implementation of eHealth prevention interventions.

Contributions To The Literature
Our trial is among the rst to study the implementation of an eHealth HIV prevention intervention.
Comparing two competing implementation strategies will contribute to knowledge about scaling up future eHealth interventions.
Our research is among the rst to use a cluster-randomized trial for HIV prevention science.
Identifying clusters for randomization based on the local density of the target population is innovative.

YMSM, HIV, and STIs
Young men who have sex with men (YMSM), ages 18-29, account for nearly 70% of all new HIV diagnoses among adolescents and young adults in the United States,(1) with HIV prevalence estimated at 13.6% in 18-24 year-olds and 18.5% in 25-29 year-olds.(2) HIV infections among YMSM are almost entirely transmitted via unprotected sex, (3)(4)(5)(6) de ned here as sex with neither Pre-Exposure Prophylaxis (PrEP) nor condoms. Sexually Transmitted Infection (STI) prevalence is also high among YMSM, (7)(8)(9) and STIs play an important role in increasing HIV transmission. (10)(11)(12) In fact, the Centers for Disease Control and Prevention (CDC) estimates a rectal STI causes a threefold increase in the per-act risk of HIV transmission during receptive anal sex. (13) Bridging the Research-Practice Divide Despite increased HIV risk, few interventions in the CDC Compendium of Evidence-Based Interventions (EBIs) and Best Practices for HIV Prevention Programs(6) are focused on YMSM. (14) The current arsenal of behavioral EBIs primarily includes face-to-face individual and small-group programs, (6) and their reach has been limited by economic and structural barriers to implementation. (15)(16)(17)(18)(19)(20) EBIs have typically been delivered through practice settings such as health departments and community-based organizations (CBO), (21)(22)(23) and evidence suggests interventions are not always implemented with delity, (15,16,24) which can produce a "voltage drop" in effectiveness. (25) eHealth approaches represent a critical modality for engaging YMSM and delivering intervention content while overcoming barriers to access (e.g., geography (26,27)) and circumventing delivery challenges (e.g., delity (28) Additional content is provided in two booster sessions three and six months after the main intervention. Table 1-Content of the KIU Intervention provides an overview of subject matter covered within each module of the intervention as well as depicts the structured breaks between modules. In Your Community Candid interviews with young gay and bisexual men about their communities, family, sex, and relationships that situate these relationships as important aspects of health. Similar interviews on various topics appear throughout the program.

Hooking Up Online
Three comic book vignettes about meeting men online for sex that focus on identifying factors that may lead to increased HIV risk, such as mood and substance use, as well as preparatory strategies to reduce those risks.

With Friends
The rst chapter of a scripted soap opera that highlights the risks of making assumptions around HIV status and monogamy and promotes positive norms around getting regular HIV testing and utilizing prevention strategies. Remaining chapters appear throughout subsequent modules.

8-hour break 2 In Bars and Clubs
An interactive game that addresses the consequences of excessive alcohol consumption and drug use, as well as decisional balance around condom use.
On Dates An animated story that explores how power dynamics in a dating relationship can affect sexual risk taking. Interactive risk calculators that demonstrate differing levels of HIV and STI risk based on various sexual behaviors.

8-hour break 3
In Relationships An interactive animated story and supporting videos that model using good communication skills in relationships to help meet one's sexual, emotional, and health needs.
In the Future A goal-setting activity that helps participants identify ways to meet their sexual, emotional, and health needs and troubleshoot obstacles to achieving those goals.

3-month break 4 Knowing Your Status
New videos and activities that focus on the importance of regular HIV testing in combination with prevention strategies like condoms and PrEP while adding additional layers of nuance, such as sexual pleasure and preventing condom use errors. There is also a check-in on participants' goals.

AIMS
In this manuscript we describe the design of the KIU 3.0 trial, which has the following speci c aims: Aim 1: Compare two implementation strategies using a cluster-randomized trial (CRT). For Strategy 1, CBOs applied for funding to deliver KIU as a service. Strategy 2 is a direct-to-consumer (DTC) model wherein research staff at Northwestern University will recruit participants nationally through online advertising campaigns and manage their engagement.
Aim 2: Examine adoption characteristics that explain variability in implementation outcomes. We will seek to explain variability in implementation success across counties by conducting mixed-methods research (40,41) on the domains from the consolidated framework for implementation research (CFIR).(42) Data on CFIR characteristics will be collected through administrative data, surveys, and teleconference interviews with key stakeholders.
Exploratory Aim. In addition to our 2 speci c aims, we will explore sustainment of KIU at the completion of the study. CBOs will be provided with materials to facilitate applying for external funding to continue to provide KIU after study completion, including an Impact Tool (43) to estimate local impact and costs. We will also examine factors that predict applying for funding and ongoing sustainment.(44) For Strategy 2, we will examine options for ongoing sustainment of the DTC model.

Cluster-Randomized Trial
Aim 1: Compare two implementation strategies using a CRT.

Sample Frame -County Selection and Randomization
For our sample frame, we selected counties with an estimated number of YMSM greater than 1,500(45) (N = 113 counties). Counties in which KIU has been studied previously were excluded. We randomized 66 counties 2:1 to the CBO and DTC strategies, respectively. Our team opted to randomize 2:1 :: CBO:DTC because we expected that some counties would not have any CBOs apply, or if they applied, some counties' applicant CBOs would not meet minimal requirements for implementing KIU. Further, we randomized 22 counties to receive the KIU DTC strategy because we expected no selection loss(46) from counties where we will advertise online directly to YMSM. KIU will be delivered in each county for two years. test negative for HIV at the time of registration, 2) report that they are assigned male at birth, 3) identify their current gender identity as male or identify as non-binary, 4) are between the ages of 18-29 at registration, 5) report CAS with a male partner in the previous six months, and 6) are not on PrEP, have been on PrEP for less than six months, or report missing a PrEP dose in the previous six months.
Participants will be recruited into KIU using different strategies based on the arm of the study into which they are recruited. For the CBO arm, participants will be recruited at CBO sites after testing negative for HIV. CBO staff will explain KIU and register interested YMSM, who will receive an email with a link that will take them to the application. Once there, participants will complete a baseline survey where their eligibility for the KIU research surveys is veri ed. In the DTC arm, participants will be recruited primarily through social media, geospatial dating apps, and supplemental approaches (e.g. referrals and print ads). Participants will click through or enter a URL to complete an initial screener for eligibility. When a participant meets all of the other indicators for eligibility, they will be sent an at-home HIV test kit to verify the HIV negative enrollment criterion as well as materials to self-administer swabs and a urine collection cup for oral, urethral, and rectal gonorrhea and chlamydia. See the section entitled DTC Recruitment in Additional File 1 -Supplemental Information for additional details on how remote recruitment, screening, and testing will take place in the DTC arm.

Implementation Strategies -Direct to Consumer Arm -Incentives
In the DTC arm, participants will receive a $10 or $25 pre-loaded virtual Visa gift card for completing the rst three sessions of the intervention. Incentive amount is randomized by county in order to determine if a threshold amount must be met for incentives to be effective. DTC participants will also receive a discount code for an online store that sells adult toys and products. Participants will be entered into an e-ra e for various prizes donated by vendors as an incentive for completing the booster content at 3-and 6-months post-intervention. This strategy is based on those successfully used by CBO partners in previous KIU service implementations to keep their clients engaged.(34) Participants across both arms of the trial are entered into ra es for $200 gift cards when they complete research surveys at baseline, 3, 6, and 12 months postintervention.

Measures
The RE-AIM model (47,48) broadly guided our outcome measurement framework. Aim 1 will use quantitative data on Reach, Effectiveness, and Implementation, while Aim 2 will pull from mixed-methods data on Adoption and qualitative data to explore Maintenance and Sustainability. Outcomes will be measured across     Our primary outcomes are public health impact (PHI), de ned as reach X effectiveness (47,49,50), and cost per infection averted. (54,55) For this trial, PHI will be assessed by (1) reach into the county's YMSM community, weighted by HIV risk, and (2) effectiveness at reducing HIV risk. (47,49,50) By measuring individual-level change on these modi able factors, our index of PHI will allow for heterogeneity in response to KIU across individuals, race and age groups, counties, and implementation condition. Effectiveness will be marked by the estimated change in that person's risk for HIV from baseline to follow-up surveys, determined by observed changes in target risk behavior: CAS, STIs, and adherent PrEP use, all of which have major impacts on HIV transmission in MSM. (56)(57)(58) We will base our modeling of HIV risk on published singleexposure probabilities and account for multiple exposures using binomial modeling.(59) Because these are individual-based measures, we will use two-level mixed-effect modeling.(60) Measuring cost is described below, and from these we will compute cost per infection averted, which is analogous to the measure used by CDC to decide which effective HIV prevention intervention would be supported as part of high impact prevention. (54,55) We will follow established guidelines for collecting cost data and conducting economic evaluations and will conduct cost analyses for each arm from the perspective of the healthcare sector. (52,53) We will follow a micro-costing approach, a technique in which all inputs consumed in an intervention are identi ed and quanti ed in detail and then converted into scal terms to produce a cost estimate.
Data Analysis -Compare CBO and DTC Implementation Strategies We will employ Multivariable Generalized Linear Mixed Models (GLMM) for our analyses, as they account for nested individual observations within counties and are commonly used in CRTs. (53,61) The multivariable aspect of the model will allow for the control of factors that are unbalanced between arms either because they were not accounted for in the randomization process or because they may have become unbalanced due to loss to follow-up. To account for potential selection bias in constructing an e cient (1 degree of freedom) adjustment for measured differences in county-level covariates, we will adjust for the linear combination forming the rst canonical covariate that maximally distinguished the sites in the two arms. (46,62) Similar to propensity score analysis, we will also formally include the model that predicts selection effects.(63) We will estimate standard errors using nonparametric bootstrapping techniques within the multivariable framework treating arm as a dichotomous indicator. (53) The differences between the intervention arms will be evaluated by examining the statistical signi cance of the level-2 (i.e., cluster level) dichotomous indicator for the intervention arms. We will estimate and compare the difference in predicted mean cost per participant between the arms, and will also use those estimates of mean cost to characterize the cost per HIV infection averted in each arm of the study.
For cost-effectiveness analysis, parameters obtained from bootstrapping will be used to estimate costeffectiveness acceptability curves which will indicate the probability that either intervention is a good value for different willingness-to-pay thresholds (i.e., incremental cost per infection avoided). (64) Finally, all analysis of outcomes will conform to best practices in analysis of randomized trials, including intention-totreat analysis and sensitivity analysis of missing data and multilevel multiple imputation in order to examine the potential effect of missing responses on the results. ( know the circumstances where one is more valuable than the other. By evaluating both strategies head-tohead in this large trial, we will be able to discern their overall effectiveness and identify key moderating factors (51,86) that could help boost and shape future dissemination and implementation of eHealth HIV prevention interventions (87,88) and contribute to greater scalability, reach, and public health impact.
Our trial will contribute to implementation science, with a particular focus on eHealth. Effective implementation has been described as the greatest challenge to HIV prevention,(89) yet there has been insu cient research testing implementation strategies that will ensure effective interventions get to the right individuals at the right time in the right dose. (15,16,20) While prior implementation science has compared capacity-building approaches for individual and small-group prevention programs, (24,(90)(91)(92) there has been little-to-none on eHealth HIV prevention. (30,31,93,94) The proposed study will lead the way in understanding the implementation of eHealth HIV prevention interventions to ensure the promise of cost-effective scalability is realized. (95) Abbreviations This protocol was approved by the IRB at Northwestern University, IRB reference number STU00207476. The protocol is currently on version 1.9. YMSM and CBO staff participants will be informed of the aims of the study as well as data protection; all participants will be consented to participate in our trial. Reports derived from our trial will be delivered in aggregate such that participants cannot be identi ed.

Consent for Publication
Not applicable.

Availability of Data and Materials
The datasets generated and/or analyzed during this trial will be available from the investigators upon reasonable request.

Competing Interests
The authors declare that they have no competing interests. Authors' Contributions BM is the principal investigator of the trial, and led the design of the study, the research team, and data collection; he participated in the writing and editing of the manuscript. JPJ wrote the abstract and sections on the county selection and RFP processes, CBO study status, and discussion; he also acted as project lead in constructing the manuscript and provided substantial editorial support across all sections. NB provided content and editorial support for the RFP process and CBO study status and contributed to the design of the study. KM and KLM co-wrote all DTC sections; KM contributed to the design of the study. GS and EB co-wrote the sections on participant inclusion criteria, measures, and data analysis as well as constructed measures tables. BRS and AS contributed content and editorial support to the data analysis and measures tables; BRS contributed to the design of the study. DHL and RS co-wrote the sections on the KIU intervention background and intervention delivery. JDS and CHB contributed to the design of the study and provided editorial support for the data analysis and measures sections of the manuscript. AD wrote the section on CBO training. BPL and PJ contributed to the design of the study. All authors read and approved the nal manuscript.