A Novel Process to Recruit Community Partners for a Hybrid Implementation-Effectiveness Study

Creating strong partnerships with community organizations is essential to testing implementation of evidence-based interventions. Given the time and investment needed build relationships, partners are often chosen based on convenience, rather than capacity or diversity. Streamlined processes are needed to identify qualied, diverse, and invested partners to conduct community-based implementation research. Currently there is a gap in the literature on effective and ecient methods for recruiting implementation partners. The goal of this paper is to ll that gap by describing a novel process for identifying a diverse group of community organizations to participate in implementation research.

Research on implementation of evidence-based public health interventions requires partnerships with community organizations, but the literature offers little guidance on how to identify partners systematically Using a novel Request for Partners (RFP) process we addressed several challenges that researchers face recruiting community partners including extended timelines and mismatch of expectations The RFP process improved partner diversity, helped build a network of partners interested in research, and attracted additional funding for implementation Publishing methods for recruitment of implementation research partners is crucial for increasing research transparency and will move the eld towards developing best practices that can improve research quality Background Research has indicated that strong partnerships between community organizations and research institutions are crucial for successfully implementing evidence-based interventions (1)(2)(3). In order to progress to the implementation, e cacious public health interventions must be tested in their intended context with delivery by appropriate community organizations rather than researchers (4)(5)(6). Strong relationships with community organizations that can both implement the intervention and participate in research are crucial to this process. While there is a breadth of research on the facilitators, barriers, and outcomes of community-academic research partnerships, there is limited literature detailing the characteristics and processes of forming these partnerships in a systematic way (7,8). This is likely because most partnerships are formed based on long-standing relationships and networking rather than a systematic process (9)(10)(11).
There are many factors that need to be considered when recruiting community organizations to engage in implementation research including organizational (e.g., budget, sta ng) and individual factors (e.g., knowledge, opinions, expectations toward research) (12). Despite recommendations for developing rationale, criteria and procedures for adding new community partners (13), this detailed documentation of how these decisions are made is often lacking in the literature. Having a systematic process for partner recruitment would allow for greater partner diversity as well as improve transparency in how communityacademic partnerships are established which may ultimately strengthen these partnerships. Furthermore, having a diverse group of implementation partners allows for a better understanding of external validity of interventions and documentation of a variety bene cial adaptation strategies (14).
When researchers engage new community organizations for the purposes of research, the literature indicates that extended timelines are common; speci cally time is needed develop trust, identify champions and manage con icting agendas with existing initiatives (15,16). Our research team previously conducted an pragmatic e cacy trial of a mobile produce market model (17), the Veggie Van. Working with one implementing partner, it took 38 months and outreach to 53 potential community sites (e.g., community health clinics, churches, schools) in order to recruit 12 nal community sites to host the Veggie Van (18). The extended timeline led to site drop-out and ultimately only eight sites ended up hosting a mobile market. In re ecting on the process, the research team found that the prolonged timeline, site drop-out and subsequent lower-than-expected participant recruitment was due to a mismatch between research and community site timelines and a lack of understanding of randomization and/or an unwillingness to be randomized (18). Despite recruitment challenges, we found that the Veggie Van increased fruit and vegetable consumption among customers at the community sites (17) and wanted to test its effectiveness when implemented by multiple partners across a variety of community sites.
In order to conduct a hybrid implementation-effectiveness study of the Veggie Van model, we needed a process for identifying implementation partners and community sites that would overcome the challenges faced during the previous e cacy trial. Given the lack of literature on recruiting community organizations for implementation research, we developed a novel Request for Partners (RFP) process which was designed to streamline partner recruitment and ultimately produce a well-quali ed, diverse and invested group of implementation partners. The present paper lls a gap in the literature by provided a detailed description of our chose partner recruitment method as well as discussing challenges and bene ts.

Overview of Veggie Van Study
The goal of this effectiveness-implementation study is to understand if mobile markets that implement the Veggie Van model can impact fruit and vegetable intake and other related health outcomes in lowerincome and underserved communities. In addition, we seek to understand how easy or hard it is for community organizations to implement the Veggie Van model. For the current study, the goal was to recruit eight implementation partners to operate mobile markets following the Veggie Van model at four community sites each (32 communities total). Prospective partners were community organizations with a mission to improve food access and/or serve populations with limited access to food to start or expand mobile markets using the Veggie Van model. The Veggie Van model is designed to address multiple dimensions of access to fresh produce by offering a variety of fresh, high quality fruits and vegetables at a reduced cost. Produce is sold at convenient locations (i.e., community sites) that are already serving the target population of low-income families through complementary services (i.e., health clinics, community centers, etc.) (19). A key component of the Veggie Van model is to encourage customers to purchase a bundle of produce (multiple items for a set price) rather than just 1 or 2 items separately. Cooking demonstrations, recipes and nutrition education are also available at the market to help customers better use the produce in their bundles.
In recruiting implementation partners, the research team was looking to identify organizations that were either not currently running a mobile market or had limited experience running a mobile market. We were also willing to work with existing markets that were interested in expanding and adopting the Veggie Van model. Each of the selected implementation partners would receive technical assistance for operating a mobile market program following the Veggie Van model and access to our online toolkit. The toolkit provides step-by step instructions and resources for running a mobile market following the Veggie Van model. Chosen partners would receive funding (up to $50,000 each) to offset the cost of adopting the Veggie Van model and participating in study-related data collection. In turn, each of the eight partners would be asked to identify four community sites that would host there mobile market. Two of these sites would be randomized to be market sites, launching a mobile market shortly after baseline data collection, while the other two would engage in a year-long community engagement and planning process with the goal of hosting a market after a 1-year follow-up data collection period. As such, the team decided an RFP process would help us identify the most appropriate candidates.

Request for Partners Development Process
The RFP process was developed to mimic a process familiar to many community organizations (i.e., applying for a grant) and was informed by our teams previous experience with using an application and selection process to identify research partners for food systems work (20,21). The rst step in the RFP process was to determine the criteria that we were looking for in our implementation partners. Next, a selection committee external to the research team was formed to guide the RFP development and selection process.

Establishing Inclusion and Selection Criteria for Implementation Partners
Our research team decided at the time of writing the grant that potential implementation partners should be limited to organizations that primarily serve an urban population, as the Veggie Van model had been previously tested in an urban area. We also limited the study region to 20 states in the north and southeast to allow our research team (based in New York and North Carolina) to more easily support the selected partners. In order to inform the speci c selection criteria, we conducted key informant interviews with 21 established mobile markets. A detailed description of the key informant methods is described elsewhere (cite key informant paper).
Using the ndings from key informant interviews and the research team's collective experience with operating mobile markets, community engagement, and food systems work, we created the initial RFP questions and selection criteria. Selection criteria included: 1.) Organizational capacity for running a mobile market; 2.) Availability of partners and community-level resources to support mobile market work; 3.) Previous experience with food access programs; 4.) Past success engaging and providing services for lower-income community members; and 5.) Potential for longer-term program sustainability. As our goal was to ensure the RFP was open to organizations that were new to running a mobile market, we indicated in the RFP that "the intent of this program is to provide training and technical assistance, thus successful partners do not necessarily need to be strong in every one of the above areas. Partners will be chosen to represent a mix of strengths across these domains."

Creation of a RFP Selection Committee
In order to select RFP committee members, the study team rst identi ed the areas of expertise that aligned with the criteria that applicants would be evaluated on, including local food aggregation and sales, logistics, mobile market operations, community engagement and equity. The study team used preexisting connections to identify individuals that demonstrated expertise in these areas and all but one accepted, instead recommending a member of his team that he felt would be well-suited for the role. The nal selection committee included ve individuals from different areas of the United States and each received an honorarium of $1,000 for helping to develop the RFP guidelines and participating on the selection process.

Create the RFP Guidelines
The full RFP guidelines can be found in Additional File 1. Brie y, the RFP guidelines indicated that the responsibilities of the implementation partners for data collection and intervention delivery would include: 1.) Identify four community sites to be randomized; 2.) Work with the community sites and study team on community engagement efforts; 3.) Run a mobile market, following the Veggie Van model, for at least one year at the 2 selected mobile market sites; 4.) Work with the study team to collect evaluation measures; 5.) Share de-identi ed customer-level sales data 6.) Track study expenditures and; 7.) Participate in collection of implementation measures (i.e., qualitative interviews, market reach, delity measures). The RFP indicated that selected partners would receive the following bene ts: 1.) Funding up to $50,000 to offset the cost of running a mobile market according to the Veggie Van Model and participating in ongoing evaluation efforts; 2.) Access to the online Veggie Van toolkit; 3.) Training and ongoing technical assistance; 4.) A License for Farmers Register Mobile Market point-of-sale software and an iPad to run the software; 5.) Access to data and reporting related to mobile market evaluation outcomes; and 6.) Assistance with developing a sustainability plan and identifying program funding sources.

RFP Release
The proposed and actual timeline for the RFP process is shown in Table 1. The RFP was released on September 3, 2018 and was advertised through a variety of food-related list serves, grant databases and professional contacts of the research team and selection committee ( Table 2). We also sent personal invitations to the 17 organizations that were not eligible for the key informant interviews because they had not been in operation for at least 2 years. After the release, potential applicants had until September 11, 2018 to submit clarifying questions and we held two informational webinars where questions were answered by the research team. The most common questions were seeking clari cation on how to describe their target market, how urban regions are de ned, how to demonstrate need (i.e. low-income or food insecure) in the areas they intend to serve, eligibility criteria for organizations, how to choose community sites to host the mobile market, use of funding, the Veggie Van model, timeline, and study procedures. A webinar recording and frequently asked questions document was posted online after the webinar. As we wanted organizations with varying levels of experience with mobile markets and/or grant writing to feel comfortable applying, we emphasized that we are available to answer any questions and support them through the process. Note: While these were o cial published deadlines, for the intent-to-apply potential applicants were told that although form submissions would be accepted after the deadline, preference would be given to those that applied by the deadline. For the Phase 2, we worked with organizations on a case-bycase basis to ensure that the deadlines were not a barrier to applying.
*A second webinar was held because the rst one failed to record In order to be considered for the RFP, community organizations were asked to rst complete an online "Intent to Apply" form (Round 1). Next, we noti ed applicants whether they were invited to complete a full application (Round 2). Finalists were selected from among the full applications and invited to an inperson training and selection process (Round 3). After the selection process, we asked nalists to submit budgets for review by the selection committee (Round 4). Lastly, nalists and those not selected were noti ed of their status.
Intent to Apply (Round 1) Interested applicants were asked to complete a brief online submission form and provide the following information: type of organization (e.g., non-pro t, hospital), location, area served (e.g., urban), mission, programs offered, target market, experience with running a mobile market, organizational reach, and the operating budget for the organization and mobile market (if applicable). For this initial round, our goal was to remove organizations that clearly did not qualify so that they would not waste time completing a full application and to limit the number of applications that the selection committee would need to review. We also wanted to ensure we had enough applications before we proceeded to the full review. We expected that around 20 organizations would be invited to submit a full proposal.
The research team reviewed the applications and excluded applicants that did not meet the eligibility criteria (i.e., predominately urban, eligible within study region). Next, we removed any organizations whose missions were not aligned with the Veggie Van model (i.e., not food access related or not serving populations with limited food access) or that had been operating a mobile market for more than two years. At this initial stage, all determinations on whether organizations met inclusion criteria were made by the research team only. In order to reduce the appearance of bias, the research team refrained from making determinations about any organizations with which they had a previous relationship and instead asked the selection committee to review these organizations during round 2. This decision was made in to avoid the perception that the study team was showing favoritism to one organization over another.

Full Application (Round 2)
Any applicants that were not excluded in the rst round were invited to submit a full application. The narrative portion of application was limited to ve single-spaced pages. Applicants were asked to answer eight questions: 1.) Describe your organization's mission or purpose; 2.) Describe the programs and services your organization currently offers; 3.) What four new community sites do you propose to work with? (letters of support required from each one); 4.) Describe your plans for community engagement with potential mobile market customers; 5.) Who at your organization will be responsible for running the mobile market program? (organizational chart required); 6.) Detail any resources which your organization currently has access to that could make a mobile produce market program successful; 7.) How do you propose using the money you receive for this program? 8.) How do you expect to continue to run the mobile market after the initial funding period?
All received applications were sent to the selection committee for review. Each application had two reviewers, one of which served as the lead reviewer. Each of the ve selection committee members reviewed approximately 6-7 applications and shared their assessments with the rest of the selection committee. Reviewers were asked to declare any con icts of interest and were not allowed to review any application with which they had a con ict. Reviewers were asked to complete a scoring sheet for each application. The scoring sheet asked them to rate each organization on a scale of 1 to 5 (1 = poor, 5 = excellent) on each of the ve selection criteria described above. We averaged the scores from both reviewers and weighted them so that each organization would have a total possible score of 100. Each criteria was weighted based on the research team and selection committee's determination of their importance as follows: capacity (30%), partners and resources (20%), previous experience (15%), community engagement (20%) and program sustainability (15%).
The combined and weighted scores for each applicant were rank-ordered. To account for differences in scoring style, each reviewer was asked to indicate the top two applicants they wanted to see proceed to the nal round. The goal at this stage was to identify 12 nalists to advance to the next phase. The selection committee met as a group with a member of the research team that facilitated the discussion.
The two most highly-rated applicants advanced to the next round without objection. Committee member discussion focused on applicants for which there was more than a 10-point discrepancy in scores between reviewer 1 and reviewer 2. Throughout the discussion, reviewers had the option to adjust (increase/decrease) or keep the same score they originally awarded. Once scores were recalculated, a clear divide emerged between the top 10 applicants and the rest of the applicants. However, two lowerscoring applicants were selected to proceed to the next round because they received a "top two" designation by at least one reviewer.

In-Person Training and Selection Process (Round 3)
Finalists were invited to attend the in-person selection process and training to learn more about the Veggie Van model, the Veggie Van study, and the requirements of being a partner. The goal of this process was not only for the selection committee to meet each applicant and to determine if they were a good t for the study, but also for the nalists to determine if the study was a good t for their organizations. Prior to attending the in-person training session, nalists were asked to submit a budget proposal (maximum $50,000) detailing how they intend to spend the funds if awarded. They were also asked to indicate within the budget any in-kind sources that they had to support the project.
A one-day training and selection meeting was offered in conjunction with an optional 2-day Mobile Market Summit, also hosted by our team but open to any mobile market practitioner, funder or researcher wishing to attend. The research team paid for all expenses for the training as well as ight and hotel for up to two members from each of the nalist organizations. The training included several sessions providing an overview of the study and partner requirements with ample opportunity for nalists to ask questions. The training day also included 10-minute Power-Point presentations from each nalist, divided into two groups. Finalists from smaller, less experienced organizations all presented in the morning and nalists from larger, more experienced organizations presented in the afternoon. After each set of presentations, nalists met privately with the selection committee and the committee had the opportunity to ask questions about the applicants' presentations, applications, and budgets. Each meeting was limited to 10 minutes with the selection committee. However, each nalist had the opportunity to sign up for one-on-one technical assistance meetings throughout the day with Veggie Van team experts and selection committee members. The study training closed with a budget session to provide guidance to nalists on developing their nal budgets.
Based on discussions with the selection committee that highlighted areas in which applications were unclear or lacking, nalists were asked to focus on certain topics in their presentations. Those topics included: 1.) Organizational capacity, sta ng structure (leadership, volunteers, paid support staff; etc.) and plans for e cient operations and data collection; 2.) Past experience serving low-income individuals and food access programs; 3.) Partnerships and resources that would be utilized to ensure the success of the mobile market; 4.) A description and justi cation of the community sites they proposed to work with; and 5.) Their long-term sustainability plan, speci cally for the mobile market.

Final Selection Process (Round 4)
After the conclusion of the in-person training and selection process, nalists were asked to revisit their budgets based on new information learned at the training and feedback from the selection committee.
Finalists submitted a nal budget three weeks after the training to be reviewed by the selection committee. Selection committee members submitted another round of scores for each nalist based on the in-person presentation at the training and proposed nal budget. The criteria for this round of scoring were designed to provide some equality across different organizations sizes and the selection committee was made aware of the value of having a diverse group of partners (not just the largest or most experienced). Speci cally, we asked each selection committee member to rate each on four areas: 1.) Preparedness and completeness of presentations; 2.) Partnerships and community sites; 3.) Responses in the Q&A session (post-presentation); and 4.) Final budget. Selection committee members rated each area on a scale of 1 to 5 (1 = Poor to 5 = Excellent) for each organization. One selection committee member was not able to attend the training due to a cancelled ight and reviewed the PowerPoint slides for the presentations at a later date, but did not provide scores for the Q&A session. Final scores from all section committee members were summed across all nalists.
During the course of the selection process, the study's budget was reduced, so that only 6 partners could be supported ($50,000 each, $300,000 total). However, during that time, we obtained additional funding to support two partners in North Carolina. As the funder chose speci c partners to support, the selection process continued excluding those partners from the rankings, as they would automatically receive the full funding amount. For the nal selection process, the Principal Investigator reviewed all selection committee scores and along with research team's input, made the nal selection decisions.

Community Site Selection and Randomization
During the RFP process, each applicant was asked to identify four community sites where they would like to host a mobile market and provide letters of support from those sites. During the partner selection process, the appropriateness of the sites and the strength of partnerships was rated by the selection committee. After the implementation partners were selected, they worked with the research team to develop a timeline for when each pair of sites would begin the community engagement and participant recruitment process. Each partner choose which two sites would be available to start rst and they became a pair for the purposes of randomization. In some cases, the partners had to identify new community sites due to timeline mismatch or changes in the site's ability to host a mobile market. Two months before each pair was scheduled to start the community engagement process, each pair of sites was randomized to be a market site or 12-month planning site (with option to start a market at the end of 12 months). Once randomized, the implementation partner was responsible for communicating to their sites the proposed timeline and responsibilities. The research team provided suggested communications about the study which the partner could use when communicating with their sites as well as a sample Memorandum of Understanding (MOU). After the MOU terms were nalized by the partner and the site, it was signed by the research team, partner and community site. Site randomization, community engagement and participant recruitment are ongoing and will be reported elsewhere. Figure 1 details the number of applicants received and advanced through each stage of the process. We received 59 intent to apply submissions, of which we invited 28 applicants to complete a full application. Thirteen applicants were rejected because they served predominately rural areas and 12 applicants because they had been operating a mobile market for 2 or more years. An additional 6 applicants were removed because their organizational mission and population served did not align with the Veggie Van model. Six applicants that had previous relationships with the study team were not reviewed during this round. Of the 28 invited applicants, 17 submitted full applications and 12 were invited to be nalists and attend the in-person selection process, training, and Mobile Market Summit.

Final scores and selection
After excluding the two North Carolina partners with separate funding from the process, nine nalists remained and nal selections were made by research team using the scoring provided by the selection committee. One organization was eliminated because all of the selection committee members recommended against it. The remaining eight nalists were ranked based on nal scores and a divide emerged between the top ve and the bottom three; therefore, we decided to accept the top ve which left one remaining slot. It was noted that 6th and 7th ranked were smaller in size than other nalists and had less capacity, but would increase the diversity of the study population. Thus, the PI recommended we accept both of these as provisional partners. The remaining (8th ranked) nalist was eliminated. The provisional partners were offered $25,000 (instead of $50,000) and asked to work with two community sites (rather than four). The other nalists (ranked 1st -5th ) were also invited to be study partners and offered $50,000 in funding each to support their efforts.

Discussion
There is limited research discussing how to recruit community partners for research. If partner recruitment is discussed, it is generally limited and does not provide extensive details about the process. While previous studies have used an RFP process to recruit research partners (21), we are not aware of any literature documenting the development, execution, or effectiveness of that process. To ll a gap in the literature related to partner recruitment for implementation studies, we have described, in detail, the RFP process used by the Veggie Van Study to recruit partners for a cluster-randomized hybrid implementationeffectiveness trial. The entire process took approximately 8 months to recruit 9 organizations and 32 community sites instead of the 38 months required to recruit 12 community sites for the previous e cacy study (18). In re ecting upon this process, we note several bene ts and lessons learned. Bene ts include a shorter recruitment timeline, improved understanding of partner responsibilities, better organizational t, and an expanded and more diverse partner network. We also recognize areas for improvement, including better alignment of partners' timelines to study needs and recognizing the difference between the capacity to implement the interventions versus carry out research activities.
The RFP process led to a good match between study partners and the intervention being tested (Veggie Van model) as is evidenced by the post-training survey. The multi-step application process, which included informational webinars and ongoing support allowed the partners, study team, and selection committee to evaluate organizational t. Speci cally, the one-day training and selection meeting was a unique and crucial event for clarifying expectations, ensuring organizational alignment with the Veggie Van model, and understanding the capabilities of the applicants.
The RFP process expanded the diversity of potential partners in both organizational characteristics (size/location/type) and leadership team characteristics (race/ethnicity). Ultimately a larger percentage of small organizations were selected as partners compared to large or medium-size organizations. While we did not expressly exclude larger partners, the selection committee was charged with choosing a diverse group of partners, not just the largest or most experienced organizations. However, this may have resulted in partners with less organizational capacity to engage in the research aspects of the project. When designing the RFP, the research team felt that an organization's strengths in engaging with lowerincome communities and operating a mobile market would translate into better recruitment for the study. However, we realized that community engagement for establishing and building interest for a market does not always translate into the engagement of more study participants.
One unintended bene t of the RFP process was a signi cant expansion of the research team's partner network. In addition to the new partners that were selected for the study, we identi ed many mobile markets through our initial outreach survey used to identify key informants, and through the RFP dissemination process. These potential partners were added to our listservs and many have continued to participate in networking and training events offered by our team, including the annual Mobile Market Summit. Notably they have also become engaged in subsequent research with our study team. The training and summit also catalyzed networking and information sharing among mobile markets; we are currently working with several of these partners to create a mobile market coalition to formalize this community of practitioners. Funders looking to support mobile markets have also seen the value in the RFP process and have provided additional support to fund more applicants. In addition to the funder that supported the two additional partners in the rst RFP, we will be initiating a second RFP process with the support of a new funder.
Based on our experience running this RFP process, we would recommend some adjustments to researchers looking to adopt a similar recruitment process. First, we would recommend a exible selection process that takes into consideration more than just scores. Although we had clear criteria and a scoring rubric for judges to review applications, we did not have a predetermined process for how we would use the scores to make nal selections. After speaking with the selection committee about their scoring rationale, we decided to add rankings in addition to the average scores. For example, one organization was not selected as a nal partner despite high average scores on the selection criteria because the selection committee ranked them lower due to the fact that organization was very large and experienced and may not bene t from the technical assistance and funding.
Our second recommendation would be to ensure that the RFP selection timeline matches with study capacity. All partners were accepted into the study at the same time, but limitations in the capacity of the study team necessitated that some partners could not launch their markets until almost a year after they were accepted. Although this was clearly communicated in the RFP and throughout the process, many partners were eager to launch their new market sites within the peak growing season. For similar reasons, previous implementation studies have recommended recruiting partners on a rolling basis (11); we are currently testing a modi ed RFP process where partners will be accepted on a rolling basis. Alternatively, there could be several staggered deadlines throughout the year, with only 2-3 new partners selected at each time. This would reduce the administrative burden on the research team and could also allow for replacing partners that may drop-out.

Conclusion
Overall using an RFP process for recruitment of implementation partners improves the alignment of priorities and timelines, increases transparency in partner selection, and has the potential to cultivate and strengthen academic-community partnerships for research. We anticipate that this process may also reduce the likelihood of study partner drop-out. However, and RFP may not be suitable for all types of implementation studies. While we used the RFP to identify implementation partners, who in turn identi ed community sites, researchers could also use this process to recruit sites directly. Our study provided funding for partners, as this is considered best practices for community-based research, but recognize that this may not be an option for all studies. Alternatively, providing technical assistance may also provide value to study partners. Other studies, such as Growing Food Connections (20), have successfully recruited partners using an RFP for technical assistance but not funding. At a minimum, we hope that this model will encourage implementation researchers to be more thoughtful and systematic about partner selection and publish the methods and results of their partner selection processes.

Figure 1
Organizations Included in Each Round of the Request for Proposals Process. 1 Reasons for not applying included lacking organizational capacity for the project (n=3), inopportune timing (n=3), not wanting to compete with another local applicant (n=2), not wanting to be randomized (n=1), non-response (n=1), and a mismatch of business models (n=1). 2 One applicant declined their invitation to become a nalist due to the organization going through strategic planning and the study timeline no longer aligned with their plans.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.