A Novel Process to Recruit Community Partners for a Hybrid Implementation-Effectiveness Study

doi:10.21203/rs.3.rs-96329/v1

Download PDF

Methodology

A Novel Process to Recruit Community Partners for a Hybrid Implementation-Effectiveness Study

https://doi.org/10.21203/rs.3.rs-96329/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Creating strong partnerships with community organizations is essential to testing implementation of evidence-based interventions. Given the time and investment needed build relationships, partners are often chosen based on convenience, rather than capacity or diversity. Streamlined processes are needed to identify qualified, diverse, and invested partners to conduct community-based implementation research. Currently there is a gap in the literature on effective and efficient methods for recruiting implementation partners. The goal of this paper is to fill that gap by describing a novel process for identifying a diverse group of community organizations to participate in implementation research.

Methods

We used a community-guided Request for Partners (RFP) approach to recruit implementation partners to participate in a cluster-randomized hybrid implementation-effectiveness study of the Veggie Van mobile produce market model. The multi-state RFP process included formative work to inform RFP development, creation of an outside selection committee, an intent-to-apply round, a full application round, and an in-person training and selection process. Chosen implementation partners receive a toolkit, training, technical assistance and funding to offset the costs of implementing the Veggie Van model and assisting with research. Data was collected to characterize applicant size, location and experience; pre-post surveys were conducted to understand the utility of the in-person training.

Results

We received 59 intent-to-apply submissions, of which we invited 28 organizations to complete full applications: 17 submitted applications and 12 finalists were invited to the in-person training and selection process. The RFP process took approximately eight months to recruit nine organizations and 32 community sites across five states with a range of size and experience. The process increased understanding of the intervention and implementation partner responsibilities; 63.6% of finalists reported being very to extremely familiar with the Veggie Van model post-training compared to 28.6% pre-training. Only one finalist dropped out (due to significant organizational change).

Conclusions

An RFP process is familiar to even small community organizations that compete for grant funding, but may not have prior research experience. This RFP process streamlined partner recruitment timelines, increased partner diversity and cultivated community among mobile market organizations. It may also improve research transparency, study completion and intervention fidelity.

Health Economics & Outcomes Research

Implementation Study

Partner Recruitment: Community-based Research

Mobile Markets

Request for Partners

Research on implementation of evidence-based public health interventions requires partnerships with community organizations, but the literature offers little guidance on how to identify partners systematically
Using a novel Request for Partners (RFP) process we addressed several challenges that researchers face recruiting community partners including extended timelines and mismatch of expectations
The RFP process improved partner diversity, helped build a network of partners interested in research, and attracted additional funding for implementation
Publishing methods for recruitment of implementation research partners is crucial for increasing research transparency and will move the field towards developing best practices that can improve research quality

Research has indicated that strong partnerships between community organizations and research institutions are crucial for successfully implementing evidence-based interventions (1–3). In order to progress to the implementation, efficacious public health interventions must be tested in their intended context with delivery by appropriate community organizations rather than researchers (4–6). Strong relationships with community organizations that can both implement the intervention and participate in research are crucial to this process. While there is a breadth of research on the facilitators, barriers, and outcomes of community-academic research partnerships, there is limited literature detailing the characteristics and processes of forming these partnerships in a systematic way (7, 8). This is likely because most partnerships are formed based on long-standing relationships and networking rather than a systematic process (9–11).

There are many factors that need to be considered when recruiting community organizations to engage in implementation research including organizational (e.g., budget, staffing) and individual factors (e.g., knowledge, opinions, expectations toward research) (12). Despite recommendations for developing rationale, criteria and procedures for adding new community partners (13), this detailed documentation of how these decisions are made is often lacking in the literature. Having a systematic process for partner recruitment would allow for greater partner diversity as well as improve transparency in how community-academic partnerships are established which may ultimately strengthen these partnerships. Furthermore, having a diverse group of implementation partners allows for a better understanding of external validity of interventions and documentation of a variety beneficial adaptation strategies (14).

When researchers engage new community organizations for the purposes of research, the literature indicates that extended timelines are common; specifically time is needed develop trust, identify champions and manage conflicting agendas with existing initiatives (15, 16). Our research team previously conducted an pragmatic efficacy trial of a mobile produce market model (17), the Veggie Van. Working with one implementing partner, it took 38 months and outreach to 53 potential community sites (e.g., community health clinics, churches, schools) in order to recruit 12 final community sites to host the Veggie Van (18). The extended timeline led to site drop-out and ultimately only eight sites ended up hosting a mobile market. In reflecting on the process, the research team found that the prolonged timeline, site drop-out and subsequent lower-than-expected participant recruitment was due to a mismatch between research and community site timelines and a lack of understanding of randomization and/or an unwillingness to be randomized (18). Despite recruitment challenges, we found that the Veggie Van increased fruit and vegetable consumption among customers at the community sites (17) and wanted to test its effectiveness when implemented by multiple partners across a variety of community sites.

In order to conduct a hybrid implementation-effectiveness study of the Veggie Van model, we needed a process for identifying implementation partners and community sites that would overcome the challenges faced during the previous efficacy trial. Given the lack of literature on recruiting community organizations for implementation research, we developed a novel Request for Partners (RFP) process which was designed to streamline partner recruitment and ultimately produce a well-qualified, diverse and invested group of implementation partners. The present paper fills a gap in the literature by provided a detailed description of our chose partner recruitment method as well as discussing challenges and benefits.

Overview of Veggie Van Study

The goal of this effectiveness-implementation study is to understand if mobile markets that implement the Veggie Van model can impact fruit and vegetable intake and other related health outcomes in lower-income and underserved communities. In addition, we seek to understand how easy or hard it is for community organizations to implement the Veggie Van model. For the current study, the goal was to recruit eight implementation partners to operate mobile markets following the Veggie Van model at four community sites each (32 communities total). Prospective partners were community organizations with a mission to improve food access and/or serve populations with limited access to food to start or expand mobile markets using the Veggie Van model.

The Veggie Van model is designed to address multiple dimensions of access to fresh produce by offering a variety of fresh, high quality fruits and vegetables at a reduced cost. Produce is sold at convenient locations (i.e., community sites) that are already serving the target population of low-income families through complementary services (i.e., health clinics, community centers, etc.) (19). A key component of the Veggie Van model is to encourage customers to purchase a bundle of produce (multiple items for a set price) rather than just 1 or 2 items separately. Cooking demonstrations, recipes and nutrition education are also available at the market to help customers better use the produce in their bundles.

In recruiting implementation partners, the research team was looking to identify organizations that were either not currently running a mobile market or had limited experience running a mobile market. We were also willing to work with existing markets that were interested in expanding and adopting the Veggie Van model. Each of the selected implementation partners would receive technical assistance for operating a mobile market program following the Veggie Van model and access to our online toolkit. The toolkit provides step-by step instructions and resources for running a mobile market following the Veggie Van model. Chosen partners would receive funding (up to $50,000 each) to offset the cost of adopting the Veggie Van model and participating in study-related data collection. In turn, each of the eight partners would be asked to identify four community sites that would host there mobile market. Two of these sites would be randomized to be market sites, launching a mobile market shortly after baseline data collection, while the other two would engage in a year-long community engagement and planning process with the goal of hosting a market after a 1-year follow-up data collection period. As such, the team decided an RFP process would help us identify the most appropriate candidates.

Request for Partners Development Process

The RFP process was developed to mimic a process familiar to many community organizations (i.e., applying for a grant) and was informed by our teams previous experience with using an application and selection process to identify research partners for food systems work (20, 21). The first step in the RFP process was to determine the criteria that we were looking for in our implementation partners. Next, a selection committee external to the research team was formed to guide the RFP development and selection process.

Establishing Inclusion and Selection Criteria for Implementation Partners

Our research team decided at the time of writing the grant that potential implementation partners should be limited to organizations that primarily serve an urban population, as the Veggie Van model had been previously tested in an urban area. We also limited the study region to 20 states in the north and southeast to allow our research team (based in New York and North Carolina) to more easily support the selected partners. In order to inform the specific selection criteria, we conducted key informant interviews with 21 established mobile markets. A detailed description of the key informant methods is described elsewhere (cite key informant paper).

Using the findings from key informant interviews and the research team’s collective experience with operating mobile markets, community engagement, and food systems work, we created the initial RFP questions and selection criteria. Selection criteria included: 1.) Organizational capacity for running a mobile market; 2.) Availability of partners and community-level resources to support mobile market work; 3.) Previous experience with food access programs; 4.) Past success engaging and providing services for lower-income community members; and 5.) Potential for longer-term program sustainability. As our goal was to ensure the RFP was open to organizations that were new to running a mobile market, we indicated in the RFP that “the intent of this program is to provide training and technical assistance, thus successful partners do not necessarily need to be strong in every one of the above areas. Partners will be chosen to represent a mix of strengths across these domains.”

Creation of a RFP Selection Committee

In order to select RFP committee members, the study team first identified the areas of expertise that aligned with the criteria that applicants would be evaluated on, including local food aggregation and sales, logistics, mobile market operations, community engagement and equity. The study team used pre-existing connections to identify individuals that demonstrated expertise in these areas and all but one accepted, instead recommending a member of his team that he felt would be well-suited for the role. The final selection committee included five individuals from different areas of the United States and each received an honorarium of $1,000 for helping to develop the RFP guidelines and participating on the selection process.

Create the RFP Guidelines

The full RFP guidelines can be found in Additional File 1. Briefly, the RFP guidelines indicated that the responsibilities of the implementation partners for data collection and intervention delivery would include: 1.) Identify four community sites to be randomized; 2.) Work with the community sites and study team on community engagement efforts; 3.) Run a mobile market, following the Veggie Van model, for at least one year at the 2 selected mobile market sites; 4.) Work with the study team to collect evaluation measures; 5.) Share de-identified customer-level sales data 6.) Track study expenditures and; 7.) Participate in collection of implementation measures (i.e., qualitative interviews, market reach, fidelity measures). The RFP indicated that selected partners would receive the following benefits: 1.) Funding up to $50,000 to offset the cost of running a mobile market according to the Veggie Van Model and participating in ongoing evaluation efforts; 2.) Access to the online Veggie Van toolkit; 3.) Training and ongoing technical assistance; 4.) A License for Farmers Register Mobile Market point-of-sale software and an iPad to run the software; 5.) Access to data and reporting related to mobile market evaluation outcomes; and 6.) Assistance with developing a sustainability plan and identifying program funding sources.

RFP Release

The proposed and actual timeline for the RFP process is shown in Table 1. The RFP was released on September 3, 2018 and was advertised through a variety of food-related list serves, grant databases and professional contacts of the research team and selection committee (Table 2). We also sent personal invitations to the 17 organizations that were not eligible for the key informant interviews because they had not been in operation for at least 2 years. After the release, potential applicants had until September 11, 2018 to submit clarifying questions and we held two informational webinars where questions were answered by the research team. The most common questions were seeking clarification on how to describe their target market, how urban regions are defined, how to demonstrate need (i.e. low-income or food insecure) in the areas they intend to serve, eligibility criteria for organizations, how to choose community sites to host the mobile market, use of funding, the Veggie Van model, timeline, and study procedures. A webinar recording and frequently asked questions document was posted online after the webinar. As we wanted organizations with varying levels of experience with mobile markets and/or grant writing to feel comfortable applying, we emphasized that we are available to answer any questions and support them through the process.

Table 1

Proposed and Actual RFP Process Timeline
RFP Steps	Proposed Timeline	Actual Timeline
RFP Released:	September 3, 2018	September 3, 2018 (ongoing for about 2 weeks)
Informational Webinar	September 13, 2018	September 13, 2018, September 20, 2018*
Intent to Apply (online form) Due	September 24, 2018	September 26, 2018
Invited Applicants Notified	October 1, 2018	October 3, 2018
Full Application Due	November 19, 2018	November 20, 2018
Finalists Notified	January 23, 2019	January 16, 2019
In-Person Training and Final Selection Process	Mid-March of 2019	March 4, 2019
Partners Selected	May 6, 2019	May 8, 2019
Note: While these were official published deadlines, for the intent-to-apply potential applicants were told that although form submissions would be accepted after the deadline, preference would be given to those that applied by the deadline. For the Phase 2, we worked with organizations on a case-by-case basis to ensure that the deadlines were not a barrier to applying.
*A second webinar was held because the first one failed to record

Table 2

Dissemination Channels for the Veggie Van Request for Partners
Dissemination Channel
Veggie Van Research Team database of 77 mobile markets throughout North America Mobile Market Network Listserv
Grantstation
Grantwatch
National and Regional Food-Related Listservs (Food Planning, North Carolina Local Foods, ComFoods)
Various coalitions (e.g., Healthy Foods Coalition, Healthy Corner Store Coalition, Farmer's Market Coalition)
Healthy Food Retail Working Group, A Program of the Robert Wood Johnson Foundation’s Healthy Eating Research and the Centers for Disease Control and Prevention (CDC) Nutrition and Obesity Policy Research and Evaluation Network (NOPREN)
Food Policy Network, Center for Livable Futures at John Hopkins
Food Narrative Project
Food Interest Group (FIG)
USDA Community Food Projects office
State Cooperative Extension Offices (OH, GA, FL, DE, KY, SC, TN, VT, NY, PA, VA, WV)
North Carolina County Extension Directors
Regional Food Policy Councils and Alliances (96 over 20 states)
North Carolina State Family and Consumer Science Agents
RFP Selection Process

In order to be considered for the RFP, community organizations were asked to first complete an online “Intent to Apply” form (Round 1). Next, we notified applicants whether they were invited to complete a full application (Round 2). Finalists were selected from among the full applications and invited to an in-person training and selection process (Round 3). After the selection process, we asked finalists to submit budgets for review by the selection committee (Round 4). Lastly, finalists and those not selected were notified of their status.

Intent to Apply (Round 1)

Interested applicants were asked to complete a brief online submission form and provide the following information: type of organization (e.g., non-profit, hospital), location, area served (e.g., urban), mission, programs offered, target market, experience with running a mobile market, organizational reach, and the operating budget for the organization and mobile market (if applicable). For this initial round, our goal was to remove organizations that clearly did not qualify so that they would not waste time completing a full application and to limit the number of applications that the selection committee would need to review. We also wanted to ensure we had enough applications before we proceeded to the full review. We expected that around 20 organizations would be invited to submit a full proposal.

The research team reviewed the applications and excluded applicants that did not meet the eligibility criteria (i.e., predominately urban, eligible within study region). Next, we removed any organizations whose missions were not aligned with the Veggie Van model (i.e., not food access related or not serving populations with limited food access) or that had been operating a mobile market for more than two years. At this initial stage, all determinations on whether organizations met inclusion criteria were made by the research team only. In order to reduce the appearance of bias, the research team refrained from making determinations about any organizations with which they had a previous relationship and instead asked the selection committee to review these organizations during round 2. This decision was made in to avoid the perception that the study team was showing favoritism to one organization over another.

Full Application (Round 2)

Any applicants that were not excluded in the first round were invited to submit a full application. The narrative portion of application was limited to five single-spaced pages. Applicants were asked to answer eight questions: 1.) Describe your organization’s mission or purpose; 2.) Describe the programs and services your organization currently offers; 3.) What four new community sites do you propose to work with? (letters of support required from each one); 4.) Describe your plans for community engagement with potential mobile market customers; 5.) Who at your organization will be responsible for running the mobile market program? (organizational chart required); 6.) Detail any resources which your organization currently has access to that could make a mobile produce market program successful; 7.) How do you propose using the money you receive for this program? 8.) How do you expect to continue to run the mobile market after the initial funding period?

All received applications were sent to the selection committee for review. Each application had two reviewers, one of which served as the lead reviewer. Each of the five selection committee members reviewed approximately 6–7 applications and shared their assessments with the rest of the selection committee. Reviewers were asked to declare any conflicts of interest and were not allowed to review any application with which they had a conflict. Reviewers were asked to complete a scoring sheet for each application. The scoring sheet asked them to rate each organization on a scale of 1 to 5 (1 = poor, 5 = excellent) on each of the five selection criteria described above. We averaged the scores from both reviewers and weighted them so that each organization would have a total possible score of 100. Each criteria was weighted based on the research team and selection committee’s determination of their importance as follows: capacity (30%), partners and resources (20%), previous experience (15%), community engagement (20%) and program sustainability (15%).

The combined and weighted scores for each applicant were rank-ordered. To account for differences in scoring style, each reviewer was asked to indicate the top two applicants they wanted to see proceed to the final round. The goal at this stage was to identify 12 finalists to advance to the next phase. The selection committee met as a group with a member of the research team that facilitated the discussion. The two most highly-rated applicants advanced to the next round without objection. Committee member discussion focused on applicants for which there was more than a 10-point discrepancy in scores between reviewer 1 and reviewer 2. Throughout the discussion, reviewers had the option to adjust (increase/decrease) or keep the same score they originally awarded. Once scores were recalculated, a clear divide emerged between the top 10 applicants and the rest of the applicants. However, two lower-scoring applicants were selected to proceed to the next round because they received a “top two” designation by at least one reviewer.

In-Person Training and Selection Process (Round 3)

Finalists were invited to attend the in-person selection process and training to learn more about the Veggie Van model, the Veggie Van study, and the requirements of being a partner. The goal of this process was not only for the selection committee to meet each applicant and to determine if they were a good fit for the study, but also for the finalists to determine if the study was a good fit for their organizations. Prior to attending the in-person training session, finalists were asked to submit a budget proposal (maximum $50,000) detailing how they intend to spend the funds if awarded. They were also asked to indicate within the budget any in-kind sources that they had to support the project.

A one-day training and selection meeting was offered in conjunction with an optional 2-day Mobile Market Summit, also hosted by our team but open to any mobile market practitioner, funder or researcher wishing to attend. The research team paid for all expenses for the training as well as flight and hotel for up to two members from each of the finalist organizations. The training included several sessions providing an overview of the study and partner requirements with ample opportunity for finalists to ask questions. The training day also included 10-minute Power-Point presentations from each finalist, divided into two groups. Finalists from smaller, less experienced organizations all presented in the morning and finalists from larger, more experienced organizations presented in the afternoon. After each set of presentations, finalists met privately with the selection committee and the committee had the opportunity to ask questions about the applicants’ presentations, applications, and budgets. Each meeting was limited to 10 minutes with the selection committee. However, each finalist had the opportunity to sign up for one-on-one technical assistance meetings throughout the day with Veggie Van team experts and selection committee members. The study training closed with a budget session to provide guidance to finalists on developing their final budgets.

Based on discussions with the selection committee that highlighted areas in which applications were unclear or lacking, finalists were asked to focus on certain topics in their presentations. Those topics included: 1.) Organizational capacity, staffing structure (leadership, volunteers, paid support staff; etc.) and plans for efficient operations and data collection; 2.) Past experience serving low-income individuals and food access programs; 3.) Partnerships and resources that would be utilized to ensure the success of the mobile market; 4.) A description and justification of the community sites they proposed to work with; and 5.) Their long-term sustainability plan, specifically for the mobile market.

Final Selection Process (Round 4)

After the conclusion of the in-person training and selection process, finalists were asked to revisit their budgets based on new information learned at the training and feedback from the selection committee. Finalists submitted a final budget three weeks after the training to be reviewed by the selection committee. Selection committee members submitted another round of scores for each finalist based on the in-person presentation at the training and proposed final budget. The criteria for this round of scoring were designed to provide some equality across different organizations sizes and the selection committee was made aware of the value of having a diverse group of partners (not just the largest or most experienced). Specifically, we asked each selection committee member to rate each on four areas: 1.) Preparedness and completeness of presentations; 2.) Partnerships and community sites; 3.) Responses in the Q&A session (post-presentation); and 4.) Final budget. Selection committee members rated each area on a scale of 1 to 5 (1 = Poor to 5 = Excellent) for each organization. One selection committee member was not able to attend the training due to a cancelled flight and reviewed the PowerPoint slides for the presentations at a later date, but did not provide scores for the Q&A session. Final scores from all section committee members were summed across all finalists.

During the course of the selection process, the study’s budget was reduced, so that only 6 partners could be supported ($50,000 each, $300,000 total). However, during that time, we obtained additional funding to support two partners in North Carolina. As the funder chose specific partners to support, the selection process continued excluding those partners from the rankings, as they would automatically receive the full funding amount. For the final selection process, the Principal Investigator reviewed all selection committee scores and along with research team’s input, made the final selection decisions.

Community Site Selection and Randomization

During the RFP process, each applicant was asked to identify four community sites where they would like to host a mobile market and provide letters of support from those sites. During the partner selection process, the appropriateness of the sites and the strength of partnerships was rated by the selection committee. After the implementation partners were selected, they worked with the research team to develop a timeline for when each pair of sites would begin the community engagement and participant recruitment process. Each partner choose which two sites would be available to start first and they became a pair for the purposes of randomization. In some cases, the partners had to identify new community sites due to timeline mismatch or changes in the site’s ability to host a mobile market. Two months before each pair was scheduled to start the community engagement process, each pair of sites was randomized to be a market site or 12-month planning site (with option to start a market at the end of 12 months). Once randomized, the implementation partner was responsible for communicating to their sites the proposed timeline and responsibilities. The research team provided suggested communications about the study which the partner could use when communicating with their sites as well as a sample Memorandum of Understanding (MOU). After the MOU terms were finalized by the partner and the site, it was signed by the research team, partner and community site. Site randomization, community engagement and participant recruitment are ongoing and will be reported elsewhere.

Figure 1 details the number of applicants received and advanced through each stage of the process. We received 59 intent to apply submissions, of which we invited 28 applicants to complete a full application. Thirteen applicants were rejected because they served predominately rural areas and 12 applicants because they had been operating a mobile market for 2 or more years. An additional 6 applicants were removed because their organizational mission and population served did not align with the Veggie Van model. Six applicants that had previous relationships with the study team were not reviewed during this round. Of the 28 invited applicants, 17 submitted full applications and 12 were invited to be finalists and attend the in-person selection process, training, and Mobile Market Summit.

Final scores and selection

After excluding the two North Carolina partners with separate funding from the process, nine finalists remained and final selections were made by research team using the scoring provided by the selection committee. One organization was eliminated because all of the selection committee members recommended against it. The remaining eight finalists were ranked based on final scores and a divide emerged between the top five and the bottom three; therefore, we decided to accept the top five which left one remaining slot. It was noted that 6th and 7th ranked were smaller in size than other finalists and had less capacity, but would increase the diversity of the study population. Thus, the PI recommended we accept both of these as provisional partners. The remaining (8th ranked) finalist was eliminated. The provisional partners were offered $25,000 (instead of $50,000) and asked to work with two community sites (rather than four). The other finalists (ranked 1st -5th ) were also invited to be study partners and offered $50,000 in funding each to support their efforts.

Applicant Characteristics

Characteristics of the mobile market applicants that participated in each round are included in Table 3. Small organizations with an operating budget of less than $250,000 per year represented the majority of intent to apply submissions (40%); 19% were medium-sized ($250,000-$999,999) and 29% were large organizations (%1 million or more). Five of the nine selected partners (56%) were categorized as small organizations. The majority of the intent to apply submissions (78%) and the final partners (78%), were non-profit organizations. The other two partners represented a university and a small business. Among those that submitted the intent to apply, 37.3% of applicants had never run a mobile market. Of the final partners, 3 organizations (33%) had never run a mobile market. The majority (59%) of intent to apply submissions and six selected partners (66%) came from the northern region of our study area.

Table 3

Request for Partners Applicant Characteristics
Organizational Characteristics	Intent to Apply Submissions (N = 59)	Full Submissions (n = 17)	Finalists (n = 11)	Selected Partners (n = 9)
Organization Annual Budget Small: $1-$250,000 Medium: $250,001-$1 million Large: $1 million+ Unknown	40% (24) 19% (11) 29% (17) 12% (7)	47% (8) 18% (3) 29% (5) 6% (1)	55% (6) 18% (2) 27% (3) 0	56% (5) 22% (2) 22% (2) 0
Organization Type Non-Profit/Faith-based Government/Education Sector For-Profit Co-operative Hospital network	78% (46) 10% (6) 5% (3) 5% (3) 2% (1)	70% (12) 6% (1) 6% (1) 12% (2) 6% (1)	82% (9) 9% (1) 9% (1) 0 0	78% (7) 11% (1) 11% (1) 0 0
Previous Mobile Market Experience, % (n) Never < 2 years 2 + Years	37.3% (22) 28.8% (17) 33.9% (20)	23.5% (4) 3% (9) 23.5% (4)	27.3% (3) 9.4% (5) 27.3% (3)	33.3% (3) 44.5% (4) 22.2% (2)
Study Region North South	59% (35) 41% (24)	65% (11) 35% (6)	73% (8) 27% (3)	67% (6) 33% (3)
NOTE: Northern States Included Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont, New Jersey, New York, Pennsylvania, Delaware and Ohio. Southern states included the District of Columbia, Florida, Georgia, Kentucky, Virginia, West Virginia, North Carolina, South Carolina, and Tennessee.

Training Evaluation

A survey was administered pre- and post-training to collect feedback from finalists on mobile market experience and operations, the satisfaction of expectations and knowledge gained by both the mobile market summit and training. Representatives from nine of the 11 applicant organizations (n = 16) completed the 54-item pre-training survey. Pre-training survey responses indicated that the main reasons organizations chose to apply for the study was an interest in evaluating their program impact (80%), desire to reach more customers (73.3%), and to help the market become more financially sustainable (60%). Prior to the training, respondents from applicant organizations were most interested in how the Veggie Van model could help their mobile market become sustainable (93.3%).

Representatives from all 11 applicant organizations (n = 12) completed the 74-item post-training survey. Following the training, all finalists reported that the Veggie Van model would be a good (18.2%) or great fit (81.8%) for their organization. There was an overall increase in understanding of the Veggie Van model among finalists; 63.6% of respondents reported being very to extremely familiar with the model (compared to 28.6% pre-training); no respondents reported lower levels of familiarity post-training. The majority also reported a high rate of clarity around partner responsibilities after attending with 72.7% of respondents that felt the community engagement responsibilities of the study were very clear to them and 27.3% felt they were somewhat clear; no respondents felt they were not at all clear. Expectations around data collection after training ranged from somewhat clear (60%) to very clear (40%). Lastly, seven respondents provided open-ended feedback and expressed most often that they would like more information on the study timeline (n = 4), community outreach and engaging sites (n = 3), data collection (n = 2), budget (n = 2 respondents), operational details (n = 2), research process (n = 1), and organizational structure (n = 1).

There is limited research discussing how to recruit community partners for research. If partner recruitment is discussed, it is generally limited and does not provide extensive details about the process. While previous studies have used an RFP process to recruit research partners (21), we are not aware of any literature documenting the development, execution, or effectiveness of that process. To fill a gap in the literature related to partner recruitment for implementation studies, we have described, in detail, the RFP process used by the Veggie Van Study to recruit partners for a cluster-randomized hybrid implementation-effectiveness trial. The entire process took approximately 8 months to recruit 9 organizations and 32 community sites instead of the 38 months required to recruit 12 community sites for the previous efficacy study (18). In reflecting upon this process, we note several benefits and lessons learned. Benefits include a shorter recruitment timeline, improved understanding of partner responsibilities, better organizational fit, and an expanded and more diverse partner network. We also recognize areas for improvement, including better alignment of partners’ timelines to study needs and recognizing the difference between the capacity to implement the interventions versus carry out research activities.

The RFP process led to a good match between study partners and the intervention being tested (Veggie Van model) as is evidenced by the post-training survey. The multi-step application process, which included informational webinars and ongoing support allowed the partners, study team, and selection committee to evaluate organizational fit. Specifically, the one-day training and selection meeting was a unique and crucial event for clarifying expectations, ensuring organizational alignment with the Veggie Van model, and understanding the capabilities of the applicants.

The RFP process expanded the diversity of potential partners in both organizational characteristics (size/location/type) and leadership team characteristics (race/ethnicity). Ultimately a larger percentage of small organizations were selected as partners compared to large or medium-size organizations. While we did not expressly exclude larger partners, the selection committee was charged with choosing a diverse group of partners, not just the largest or most experienced organizations. However, this may have resulted in partners with less organizational capacity to engage in the research aspects of the project. When designing the RFP, the research team felt that an organization’s strengths in engaging with lower-income communities and operating a mobile market would translate into better recruitment for the study. However, we realized that community engagement for establishing and building interest for a market does not always translate into the engagement of more study participants.

One unintended benefit of the RFP process was a significant expansion of the research team’s partner network. In addition to the new partners that were selected for the study, we identified many mobile markets through our initial outreach survey used to identify key informants, and through the RFP dissemination process. These potential partners were added to our listservs and many have continued to participate in networking and training events offered by our team, including the annual Mobile Market Summit. Notably they have also become engaged in subsequent research with our study team. The training and summit also catalyzed networking and information sharing among mobile markets; we are currently working with several of these partners to create a mobile market coalition to formalize this community of practitioners. Funders looking to support mobile markets have also seen the value in the RFP process and have provided additional support to fund more applicants. In addition to the funder that supported the two additional partners in the first RFP, we will be initiating a second RFP process with the support of a new funder.

Based on our experience running this RFP process, we would recommend some adjustments to researchers looking to adopt a similar recruitment process. First, we would recommend a flexible selection process that takes into consideration more than just scores. Although we had clear criteria and a scoring rubric for judges to review applications, we did not have a predetermined process for how we would use the scores to make final selections. After speaking with the selection committee about their scoring rationale, we decided to add rankings in addition to the average scores. For example, one organization was not selected as a final partner despite high average scores on the selection criteria because the selection committee ranked them lower due to the fact that organization was very large and experienced and may not benefit from the technical assistance and funding.

Our second recommendation would be to ensure that the RFP selection timeline matches with study capacity. All partners were accepted into the study at the same time, but limitations in the capacity of the study team necessitated that some partners could not launch their markets until almost a year after they were accepted. Although this was clearly communicated in the RFP and throughout the process, many partners were eager to launch their new market sites within the peak growing season. For similar reasons, previous implementation studies have recommended recruiting partners on a rolling basis (11); we are currently testing a modified RFP process where partners will be accepted on a rolling basis. Alternatively, there could be several staggered deadlines throughout the year, with only 2–3 new partners selected at each time. This would reduce the administrative burden on the research team and could also allow for replacing partners that may drop-out.

Overall using an RFP process for recruitment of implementation partners improves the alignment of priorities and timelines, increases transparency in partner selection, and has the potential to cultivate and strengthen academic-community partnerships for research. We anticipate that this process may also reduce the likelihood of study partner drop-out. However, and RFP may not be suitable for all types of implementation studies. While we used the RFP to identify implementation partners, who in turn identified community sites, researchers could also use this process to recruit sites directly. Our study provided funding for partners, as this is considered best practices for community-based research, but recognize that this may not be an option for all studies. Alternatively, providing technical assistance may also provide value to study partners. Other studies, such as Growing Food Connections (20), have successfully recruited partners using an RFP for technical assistance but not funding. At a minimum, we hope that this model will encourage implementation researchers to be more thoughtful and systematic about partner selection and publish the methods and results of their partner selection processes.

Ethics approval and consent to participate

Evaluation of the in-person training was deemed exempt by the University at Buffalo Institutional Review Board.

Consent for publication

Not applicable

Availability of data and materials

The data used during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This research was funded by the National Cancer Institute (R37CA215232).

Authors' contributions

LL conceptualized the study. LL, SR, CK, AA, LHM, LTM, CHM and LV contributed to the design of the RFP process. CK, LV, LL, LHM, LTB and CHM conducted training for potential partners. AL and CK analyzed the data. LL, CK and AL drafted the manuscript. All authors critically edited the manuscript and approved the final version.

Acknowledgements

We would like to thank our selection committee and all of the organizations that participated in the Request for Partners process.

Haynes-Maslow L, Parsons SE, Wheeler SB, Leone LA. A qualitative study of perceived barriers to fruit and vegetable consumption among low-income populations, North Carolina, 2011. Prev Chronic Dis. 2013;10:E34.
Haynes-Maslow L, Auvergne L, Mark B, Ammerman A, Weiner BJ. Low-Income Individuals’ Perceptions About Fruit and Vegetable Access Programs: A Qualitative Study. Journal of Nutrition Education and Behavior. 2015;47(4):317-24.e1.
Kasprzak CM, Sauer, H.A., Schoonover, J.J.,Lapp, M.M., Leone, L.A Barriers and facilitators to fruit and vegetable consumption among lower-income families: matching preferences with stakeholder resources. Journal of Hunger and Environmental Nutrition. In Press.
Milat AJ, Bauman A, Redman S. Narrative review of models and success factors for scaling up public health interventions. Implementation Science. 2015;10(1):113.
Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93(8):1261-7.
Flay BR, Biglan A, Boruch RF, Castro FG, Gottfredson D, Kellam S, et al. Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci. 2005;6(3):151-75.
Drahota A, Meza RD, Brikho B, Naaf M, Estabillo JA, Gomez ED, et al. Community‐academic partnerships: A systematic review of the state of the literature and recommendations for future research. The Milbank Quarterly. 2016;94(1):163-214.
Wright MT, Roche B, von Unger H, Block M, Gardner B. A call for an International Collaboration on Participatory Research for Health. Health Promotion International. 2009;25(1):115-22.
Pinto RM, Spector AY, Witte SS, Gilbert L. Systematizing Planning and Formative Phases of HIV Prevention Research: Case Studies from Brazil, Mongolia, and Kazakhstan. Global Social Welfare. 2014;1(3):137-44.
Pinto RM, Witte SS, Wall MM, Filippone PL. Recruiting and retaining service agencies and public health providers in longitudinal studies: Implications for community-engaged implementation research. Methodological Innovations. 2018;11(1):2059799118770996.
James AS, Richardson V, Wang JS, Proctor EK, Colditz GA. Systems intervention to promote colon cancer screening in safety net settings: protocol for a community-based participatory randomized controlled trial. Implementation Science. 2013;8(1):58.
Pinto RM, Wall MM, Spector AY. Modeling the Structure of Partnership Between Researchers and Front-Line Service Providers: Strengthening Collaborative Public Health Research. J Mix Methods Res. 2014;8(1):83-106.
Seifer SD. Building and sustaining community-institutional partnerships for prevention research: findings from a national collaborative. Journal of urban health : bulletin of the New York Academy of Medicine. 2006;83(6):989-1003.
Green L, Nasser M. Furthering Dissemination and Implementation Research: The Need for More Attention to External Validity. Dissemination and Implementation Research in Health: Translating Science to Practice. 2012.
Cruz TH, Davis SM, FitzGerald CA, Canaca GF, Keane PC. Engagement, recruitment, and retention in a trans-community, randomized controlled trial for the prevention of obesity in rural American Indian and Hispanic children. The journal of primary prevention. 2014;35(3):135-49.
Whitt-Glover MC, Borden SL, Alexander DS, Kennedy BM, Goldmon MV. Recruiting African American Churches to Participate in Research: The Learning and Developing Individual Exercise Skills for a Better Life Study. Health promotion practice. 2015.
Leone LA, Tripicchio GL, Haynes-Maslow L, McGuirt J, Grady Smith JS, Armstrong-Brown J, et al. Cluster randomized controlled trial of a mobile market intervention to increase fruit and vegetable intake among adults in lower-income communities in North Carolina. International Journal of Behavioral Nutrition and Physical Activity. 2018;15(1):2.
Tripicchio GL, Grady Smith J, Armstrong-Brown J, McGuirt J, Haynes-Maslow L, Mardovich S, et al. Recruiting Community Partners for Veggie Van: Strategies and Lessons Learned From a Mobile Market Intervention in North Carolina, 2012-2015. Prev Chronic Dis. 2017;14:E36.
Leone LA, Tripicchio GL, Haynes-Maslow L, McGuirt J, Grady Smith JS, Armstrong-Brown J, et al. A Cluster-Randomized Trial of a Mobile Produce Market Program in 12 Communities in North Carolina: Program Development, Methods, and Baseline Characteristics. J Acad Nutr Diet. 2019;119(1):57-68.
Raja S, Whittaker J, Hall E, Hodgson K, Leccese J. Growing food connections through planning Lessons from the United States. In: Cabannes Y, Marocchino C, editors. Integrating Food into Urban Planning: UCL Press; 2018. p. 134-53.
Clark JK, Freedgood J, Irish A, Hodgson K, Raja S. Fail to Include, Plan to Exclude: Reflections on Local Governments' Readiness for Building Equitable Community Food Systems. Built Environment. 2017;43(3):315-27.

Download PDF

Version 1

posted

You are reading this latest preprint version

A Novel Process to Recruit Community Partners for a Hybrid Implementation-Effectiveness Study

Status:

Version 1

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Discussion

Conclusion

Declarations

References

Supplementary Files

Status:

Version 1