Process evaluation of an evidence-based weight loss program for low-income, mid-life women implemented in local health departments: Application of the RE-AIM framework

Background: Fundamental to successfully translating evidence-based interventions to real life settings with diverse populations is achieving delity to the intervention within the given public health setting. Health departments may be key in addressing the obesity epidemic given their unique position to deliver obesity prevention services in community settings. We developed the Weight-Wise II Program from four evidenced-based interventions for implementation in local health departments. For this study, we conducted a process evaluation of the implementation of the Weight-Wise II Program, an intensive evidence-based behavioral weight loss program for low-income, mid-life women. Methods: The Weight-Wise II Program, a 16-week group-based weight loss program, was implemented in six local health departments. The RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) served as a guide to conduct a comprehensive process evaluation. Process data were collected and analyzed using quantitative and qualitative methods. Results: The Weight-Wise II Program reached low-income and mid-life women, a high-risk population, and was effective in achieving modest weight loss. Attendance and self-monitoring were positively associated with weight loss. Interventionists delivered the program as intended and it was well received by participants. Planned adaptations enhanced program implementation by being responsive to participant and health department needs. Despite implementation by health department staff, the program was not routinely maintained three years post-intervention. Conclusion: Evidence-based weight loss interventions can be successfully implemented in local health departments. RE-AIM is a useful framework for systematically evaluating the process of implementation and impact of a behavioral weight loss intervention offered in public health settings. The addition of complementary implementation frameworks may help in identication of contextual factors inuencing subsequent maintenance of evidence-based interventions in public health settings.

Starting with a rigorous sampling strategy to address limitations to generalizability, we conducted a comprehensive process evaluation guided by the RE-AIM framework and including multiple qualitative and quantitative evaluation methods.
Study ndings address gaps in the literature by providing an example of how to expand the evaluation of implementation processes to include assessment of participant engagement and program sustainability, and demonstrating an approach to program implementation that considers organizational context, and integrates planned adaptations while maintaining delity.

Background
Intensive behavioral weight management interventions have been successful in changing dietary and physical activity behaviors, achieving clinically signi cant weight loss (i.e., weight loss > 5% of initial body weight) [1][2][3][4], and improving cardiovascular risk factors including prevention of type 2 diabetes [3,[5][6][7]. Successful adaptation and effective implementation of lifestyle interventions in practical settings serving low-income and minority populations are necessary to decrease obesity and reduce disease burden [8]. To this end, health departments may play a vital role in addressing the obesity epidemic [9,10] as they are uniquely positioned to deliver obesity prevention and treatment services through coordination of training and technical assistance, and engagement of stakeholders [11,12]. At the local level, health departments not only understand health issues confronting the community, but also engage the community to address health issues and provide timely, evidence-based health programs and interventions implemented by trained staff [9,13]. Moreover, some evidence suggests that individuals residing in counties with local health departments that offer population-based obesity prevention services (e.g. individual, group, or peer nutrition counseling and weight loss programs, advocacy and policy change activities supporting healthy eating and active living environments) have smaller increases in obesity risk relative to those living in areas where such services are not offered [9].
The exibility to adapt evidence-based interventions facilitates translation from research to practice through tailoring of the intervention to t participant characteristics and local resources [14,15]. Yet delity, or the extent to which an intervention is delivered as planned, must be maintained to avoid type III errors (erroneously concluding that the outcomes are a result of the intervention when the intervention is not implemented as planned), ensure reliability and validity, and document deviations and variations to program implementation [16][17][18][19]. Thus, a comprehensive evaluation plan that clearly identi es core components prior to the start of intervention and includes a systematic approach to ongoing monitoring of adaptations and program implementation [14] with routine feedback to program staff is vital to thoroughly assess delity.
We developed the Weight-Wise II intervention for implementation in local health departments from four evidencebased weight loss interventions: 1) the Diabetes Prevention Program (DPP) [20]; 2) the Dietary Approaches to Stop Hypertension (DASH) intervention tested in the PREMIER trial [21]; 3) the weight loss program of the Weight Loss Maintenance (WLM) trial [2], and 4) the original Weight-Wise Program [22]. For this study, we conducted a comprehensive process evaluation of the Weight-Wise II Program using the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework because it provides a useful approach to evaluating translational research that includes components for assessing internal and external validity [23] and our research team had particular expertise in using this framework to evaluate the public health impact of research translated to real world settings. In this paper, we describe the monitoring procedures and evaluation measures, and report process evaluation ndings.

Methods
Weight-Wise II Program-Details about the Weight Wise II Program (rationale, design, staff training, and sample characteristics) and the randomized controlled trial conducted to test its effectiveness (including sampling of study sites and intervention effectiveness) are published elsewhere [24][25][26]. The Weight-Wise II Program, a behavioral weight management intervention for low-income, midlife women, was designed for implementation in local health departments by trained interventionists from the health departments' current staff. Health departments were recruited at a meeting of North Carolina local health department directors, by mail, or through email lists to health directors, nursing directors, health educators, and health departments participating in the North Carolina Breast and Cervical Cancer Control Program (BCCCP) and WISEWOMAN (Well-Integrated Screening and Evaluation for Women Across the Nation) [24,26].
Study Sites-Each participating health department selected staff persons (i.e., registered dietitian, registered nurse, or health educator) to serve as the program interventionist. Selection of the assistant to the interventionist was at the discretion of the health department as there were no skill-based or education requirements speci ed in the research protocol. Assistants helped interventionists with activities including preparing for group sessions, shopping for foods, making reminder and follow-up phone calls, and collecting and entering data. Training for interventionists and assistants was conducted by research staff before the intervention, during six scheduled interventionists' conference calls, and via feedback as a part of intervention monitoring activities (site observation visits and audio-recorded session reviews). Pre-intervention in-person training comprised of 4 weekly group-based sessions lasting 5 hours each for a total of 20 hours, homework assignments (2 hours per weekly training session for a total of 8 hours), and online instruction (4 -6 hours). During these training sessions, interventionists were provided with an overview of the study and study materials, training in research ethics, weight management, motivational interviewing principles, group counseling and facilitation strategies, and program implementation planning. A 4-hour protocol training session for interventionists and assistants was conducted at each health department immediately prior to implementation of the Weight-Wise II Program. Interventionists received a Weight Wise II Program Protocols Manual and training on con dentiality, participant recruitment and enrollment, data collection, measuring participant's height, weight, and blood pressure, and general study protocols.
Study Participants-The target participants were mid-life, low-income women with overweight/obesity. Participant recruitment is described in detail elsewhere [26]. Brie y, participant recruitment consisted of study advertisement to health department clients and pre-screening interviews with interested women. Five templates consisting of a study brochure, yer or poster, newspaper ad, public service announcement (PSA) and a letter for health department clients were developed for study sites to use in participant recruitment. Interested women who called the health department were interviewed to assess eligibility, motivation to lose weight, readiness to start an intensive weight loss program, potential barriers to participation, and available support persons.
The overall goal of the Weight-Wise II program was to promote weight loss through decreased caloric intake and increased physical activity. To this end, participants were randomly assigned to one of two groups: special intervention or delayed intervention. During the trial, the special intervention group received an intensive 16-week group-based program with weekly sessions lasting 1.5 to 2 hours. Each group session consisted of ve segments: Check-In, during which each participant's weight was measured and previous session content reviewed; Try It, where a variety of topics and activities were covered including planning ahead for meals and identifying environmental triggers for physical activity and other support systems; Do It, which included physical activity demonstration segments; and Taste It, food tasting and demonstration segments which were offered at alternate group sessions; and Next Steps, 15 to 20 minutes of goal-setting and action planning for the next week.
The delayed intervention group received two newsletters with general health information and Weight-Wise II program updates during months 2 and 4 of the 16-week waiting period. After the special intervention group was completed, the delayed intervention group received a shorter 10-week version of the program. Table 1 outlines our operational de nitions of the RE-AIM dimensions, process evaluation measures, and data sources used to measure each dimension. Details of the assessment of each RE-AIM dimension are described below.

RE-AIM Measures-
Reach -To assess reach, interventionists screened potential participants to determine eligibility for the Weight-Wise II Program. Prescreening logs were used to determine the total number of women eligible to participate, representativeness of participants to non-participants, and reasons for non-participation. Individual participants included women ages 40 to 64 years of age who met the following inclusion criteria: 1) BMI between 27.5 and 45 kg/m 2 inclusive; 2) English speaking; 3) willing to lose 5% or more of body weight and follow recommendations for healthy dietary and physical activity patterns; and 4) family income ≤ 250% of federal poverty guidelines [26].
Effectiveness-We assessed effectiveness by measuring the impact of the intervention on primary and secondary outcomes. Weight change, our primary outcome, was calculated as the 5-month follow up weight (measured during the week of the 16 th session) minus baseline ( rst session) weight. Participants were weighed at baseline and during each weekly group session; nal weights were assessed by trained staff not involved in delivering the intervention and masked to participants' group assignment. Weight was measured in duplicate or triplicate with a SECA 770 digital scale (SECA Corporation, Hanover, MD). The average of the 2 or 3 weights (3 weights were used if the difference between the rst 2 measures was greater than 1.0 pound) was recorded to the nearest 0.1 pound.
Enactment Fidelity-Participant behaviors (secondary study outcomes) included attendance at weekly group sessions and self-monitoring of food intake and physical activity (PA). Participant attendance was recorded at each weekly group session by interventionists or assistants. Food and tness diaries were distributed to participants each week to self-monitor dietary intake (food amount, calories, and fruits and vegetable servings), type of physical activity, and daily and weekly minutes of moderate intensity PA; completed food and tness diaries were reviewed weekly by the interventionist. We assessed the number of days per week dietary intake was recorded and the number of minutes of weekly PA.
AcceptabilityAt the end of the intervention, participants were asked to complete a 9-item questionnaire to rate their satisfaction with program duration (1 = too many sessions, 2 = just about right, 3 = not enough sessions), session materials, activities, and interaction with other participants (1 = not very good to 5 = very good) and overall satisfaction with the Weight Wise II Program (1 = not very satis ed to 5 = very satis ed).
Adoption-Adoption was calculated as the number of sites eligible and interested in participating. Eligibility for participation in the Weight-Wise II Program and characteristics of interested and enrolled health departments were captured during recruitment on the Health Department Survey. Local health departments were selected using an optimized probability sampling protocol to ensure 1) the six participating health departments were representative of small, medium, and large counties; 2) no more than one health department was from the same health district; 3) a 30% racial/ethnic minority population in at least half of the health departments; 4) no more than one site with a bachelor's level health educator; and 5) no more than one health department more than 200 miles from Chapel Hill, North Carolina [24]. Implementation-Multiple measures were used to assess implementation, including training delity, adherence to program protocols; delivery delity (i.e. quality, comprehensiveness, and accuracy of program delivery); and adaptations to the Weight-Wise II Program intervention.
Training FidelityAt the beginning and completion of the in-person training session, interventionists and assistants completed an 11-item questionnaire assessing their level of con dence in: 1) knowledge of nutrition and physical activity for weight loss and cardiovascular disease reduction; and 2) ability to implement behavioral weight loss and weight management programs (1 = low to 10 = high) and indicated the estimated time spent completing online training and homework.
Adherence to program protocolsAdherence was assessed by determining the number of group sessions held by each interventionist, the session duration, and whether the interventionist followed-up with participants who had unplanned absences. The Group Session Attendance form was used to determine the number of unique group sessions held at each site; as well as to document participant attendance. Session duration (calculated from start and end times) was recorded using a Monitoring Checklist during direct observations of group sessions or using the Content Accuracy Checklist during a review of audio recordings of group sessions.
Delivery delityWe assessed delivery delity by determining the quality, comprehensiveness, and accuracy of delivery of program materials. Quality of delivery indicated how well the program was delivered overall. To determine quality of program delivery, ve of the 16 group sessions per health department were assessed using the Monitoring Checklist. Two of these group sessions were directly observed by study personnel, while three group sessions were audio-recorded for review. Research staff were trained to complete the Monitoring Checklist that was organized into sections corresponding to the format of group sessions (Check In, Try It, Next Steps, and Taste It or Do It).. Checklists were scored on Core Segments (Check In, Do It, and Next Steps) and Core Plus Segments (Check In, Do It, Taste It or Try It, and Next Steps).. Each section of the Monitoring Checklist consisted of 3 questions rated on a 5-point scale with responses ranging from 1 = not at all to 5 = completely. The three section questions were averaged to determine the section score. Subsequently, section scores were averaged to calculate the overall session score. Additionally, three open-ended questions allowed the rater to provide session summary notes and comments.
Comprehensiveness and accuracy of the program material delivered was assessed using the Content Accuracy Checklist organized into sections corresponding to the group session format. Two trained research staff reviewed the three group sessions that were audio-taped at each health department (i.e., 18 sessions totaling 27-36 hours). Comprehensiveness was de ned as the degree to which the interventionist covered program materials. Accuracy was de ned as the extent to which the interventionist provided correct information on program session topics.
Adaptations-We captured adaptations to the Weight-Wise II Program intervention using various methods prior to, during, and post-intervention implementation. The Adaptation Rationale Form was provided to each interventionist to document adaptations after review of program materials but prior to implementation. Interventionists discussed unplanned adaptations (e.g., changes in planned physical activity due to weather conditions) during weekly interventionist conference calls. The Staff Survey was administered post-intervention to identify adaptations to the program and to assess facilitators and barriers to program maintenance [25].
Maintenance -We assessed maintenance at two time points to determine the extent to which the Weight-Wise II Program was integrated into the programs and services offered at participating health departments. First, we administered a Staff Survey at the conclusion of the study which included 2 items assessing intent to implement the Weight-Wise II Program again. Subsequently, three years after program implementation, we asked local health directors of North Carolina's 85 health departments to complete an online Post-Intervention Health Department Capacity Survey assessing capacity to implement intensive behavioral weight loss programs. Three items on this survey assessed whether the Weight-Wise II Program was continued after the study, identi ed the sustained program components, and assessed the duration for continuous implementation (i.e., less than 6 months, 6 to 12 months, or more than 12 months).

Statistical Analysis
Statistical analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC) and the IBM SPSS Statistics for Windows, Version 20.0 (Armonk, NY:IBM Corp.). Each RE-AIM construct was tabulated and reported. Descriptive statistics were calculated for measures for primary and secondary outcomes and for implementation delity. We used a chi-square test to assess whether follow-up contact with participants who missed a group session varied by site. We used t test to test the association between weight loss and participant acceptability, delity measures and interventionist's characteristics, and delity measures and weight loss. A comparison of the reliability of measurements from the two research staff members who rated the Content Accuracy Checklist was conducted using the intraclass correlation coe cient (ICC) [27] on a sample of 18 audio-recorded sessions because content accuracy ratings were measured on a continuous scale. ICC scores of 0.81 to 0.90 indicate a high level of agreement between the raters.

Results
Using the RE-AIM framework, the results for our assessment of Reach include the total study sample, special intervention group and delayed control group. Results for Effectiveness, Adoption, Implementation, and Maintenance are limited to the special intervention group as the purpose of this research was to conduct a process evaluation of the 16 week-long intervention.
Reach-There were a total of 432 interested respondents, of which 213 (49%) were eligible and interested in study participation. Of these, 189 completed baseline measures and took part in the study (reach was 89%). Unavailable during group session time was the most common reason respondents (n = 11) gave for not participating in the study. Of those who were eligible and interested but did not enroll in the study (n = 24), they either missed their scheduled appointment or did not complete follow-up. Of the 189 women enrolled in the study, 126 were randomly assigned to the special intervention group and the remaining 63 to the delayed intervention control group. Participant characteristics are detailed elsewhere [26]. In short, at baseline a little more than half (53%) of study participants were African American. Participants had a mean age of 50.8 years (SD: 0.7) with an average of 13 years of education (SD: 0.2). Fewer than half (42%) were married or living with a partner.
EffectivenessDetails of Weight Wise II study effectiveness are detailed elsewhere [25]. The goals of the Weight-Wise II Program intervention were to achieve a weight loss of 10 lbs. through caloric restrictions (no less than 1200 calories per day) and at least 150 minutes of moderate intensity physical activity each week [25,26]. Table 2 summarizes weight change and measures of participant attendance and self-monitored behaviors by site. Weight change, our primary outcome, was calculated as the 5-month follow-up weight minus the weight at the start of the intervention. Of the 101 intervention participants with baseline and follow-up weights, the mean weight loss was 3.8 kg (SD = 4.9); when missing follow-up weight data was substituted with the last weight measured (last observation carried forward), the mean weight loss among all intervention participants was 3.1 kg [26]. Participants without follow-up weight assessments more often were African American, younger, took blood pressure medication, and lived with fewer adults [25]. Acceptability-Of 104 participants, 58% (n = 60) thought 16 sessions were about the right number, 32% (n = 33) believed 16 sessions were not enough, 5% (n = 5) thought 16 sessions were too many, and 5% (n = 5) did not respond. In rating the overall program, 87% (n = 91) of participants were very satis ed or satis ed, 6% (n = 6) remained neutral, 1% (n = 1) not satis ed, and 6% (n = 6) did not respond. Higher rating of program satisfaction was associated with weight loss (p =.05).
AdoptionAt the organization level, 81 county health departments and regional health districts in North Carolina within 200 miles of the research center were invited to participate in the study by completing the Health Department Survey [24][25][26]. Of the 81 potential health departments, 68 (84%) returned the survey with 25 (37%) health departments indicating they were not eligible to apply. From the remaining 43 (63%) health departments that were eligible for study participation, 30 (70%) indicated interest in participating. Eligible and interested health departments had larger county populations than those that were ineligible or eligible but not interested. Health departments that did not return the survey had smaller staff and were in smaller counties with smaller racial/ ethnic minority populations.
At the staff level, all 6 interventionists had bachelor's degrees or higher with a mean of 10 years public health work experience [24,26]. While four interventionists had previously developed, implemented, or evaluated a weight management intervention, only one had received training speci c to adult weight management. Demographic data was not collected for intervention assistants because they were selected by the health department and not assigned a role in delivery of intervention content.
Implementation -Training Fidelity:Training for Weight-Wise intervention staff included homework assignments and online instruction. The mean time to complete the two online modules was 5.3 hours, while homework assigned as part of the weekly group sessions was completed in 1.9 hours. Interventionists' and assistants' self-assessment of their con dence in knowledge of nutrition and physical activity for weight management and cardiovascular disease reduction and ability to implement behavioral weight loss and weight management programs is presented in Table 3. Thirteen pre-training and 12 post-training surveys were returned and included in the analysis. The increase in selfreported levels of con dence post-training was not statistically signi cant.
Adherence to program protocols: The extent to which the intervention was implemented at each site and overall is presented in Table 4. To further explore interventionists' differences in implementation, we assessed associations between selected interventionist characteristics (i.e., years worked in public health, years employed at participating health department, and previous experience developing, implementing, and evaluating a weight management program) and implementation delity measures of quality, accuracy, and comprehensiveness of delivering Core and Core Plus Segments. Interventionists with previous weight management experience delivered the Core Segments (Check In, Do It, and Next Steps) with lower quality (p = 0.05) compared to interventionists with no previous weight management experience. Additionally, the duration for Core Segments was signi cantly longer (p = 0.01) for interventionists without previous weight management experience.
Delivery delity: The quality, accuracy, and comprehensiveness with which interventionists delivered the Weight-Wise II Program at their sites are detailed in Table 4. The quality of program delivery averaged 3. Maintenance: Immediately after implementation of the study intervention, all but one of the six interventionists were uncertain about the intent of the health department to continue implementing the Weight-Wise II Program. Three years post-intervention, health directors of participating health departments were asked to indicate if the Weight-Wise II Program was continued after the end of the intervention on the Post-Intervention Health Department Capacity Survey. Sixty-seven percent (n = 4) of the health directors indicated that their health department did not continue implementing the program after the intervention ended. One health department continued the full 16-week program for less than 6 months after intervention completion. Another health department continued to implement components of the Weight-Wise II Program for more than 12 months post intervention. We did not elicit additional information from sites about facilitators and barriers to continued program delivery. However, one health director reported that funding was the main factor for not maintaining the program.

Discussion
We used the RE-AIM framework as a guide to conduct a comprehensive process evaluation of the Weight-Wise II Program, an intensive behavioral weight loss program for mid-life low-income women implemented in local health departments. Overall, we reached low-income, mid-life women, the Weight-Wise II Program was well received by interventionists and participants alike, was successfully implemented by health department sites, and resulted in participant weight loss. However, long-term maintenance at the setting level was low.
The successful implementation of the Weight-Wise II Program may be attributed to the exibility to adapt the program. Adaptability may make the program more acceptable but delity, the extent to which the intervention is delivered as intended [16], is more di cult to achieve when changes are made to essential components of the intervention [28]. In our case, we determined that adaptations, captured prior to and during program implementation, did not alter core elements of the program but enhanced implementation by being responsive to the needs of participants (e.g. food and physical activity preferences) and site-speci c circumstances (e.g. inclement weather, lack of cooking facilities). Similar adaptations have been made when translating evidenced-based interventions [15,29].
We had the opportunity to report maintenance 3 years post study completion and found that only two sites continued offering at least some components of the intervention. The lack of program maintenance is somewhat disappointing given that we trained health department staff who might implement such programs as a part of their job duties; and evidence of staff interest in continuing program implementation was noted on the post-intervention staff survey. Zhang and colleagues conducted an assessment of characteristics of local health department capacity associated with providing obesity prevention services [12] and found sta ng levels, proportion of state funding, employment of health educators, chronic disease and risk behavior surveillance, and development of a health improvement plan or health assessment were positively associated with obesity prevention programs. However, even when staff are available it cannot be assumed that these services are provided. In a pro le of public health educators in North Carolina, Glascoff and colleagues [30] noted that health educators with administrative roles may lack the time to deliver programs. Similarly, we found that interventionists without assistants may have lacked the time to conduct follow-up calls to participants with unplanned absences. Moreover, obesity prevention services are often not mandated thus depend on funds through sources such as private foundations and grants. These funding mechanisms are often given with short notice for grant applications and limited duration of funding [11].
It has been suggested that public health decision-makers may be more accepting of implementing interventions that have been effectively tested in similar settings and with comparable populations [31]. Based on our experience and consistent with other studies [32,33], evidence-based interventions implemented in community settings were not maintained. It appears that even in light of evidence showing intervention effectiveness, other factors at the setting level may help explain aspects of adoption and maintenance needed for program sustainability. The Consolidated Framework for Implementation Research (CFIR), comprised of constructs from various implementation theories, was developed to help researchers determine what works where and why [34]. The 5 domains (the intervention, inner setting, outer setting, individuals involved in implementation, and the implementation process) can be used to help guide assessments and explain ndings. Using a complementary framework such as CFIR, may help researchers improve the identi cation of contextual factors, capacity, and needs, to implement and subsequently maintain interventions in real-world settings.
Our study has a number of noteworthy strengths and limitations. Strengths of this study include: a rigorous sampling strategy to address limits to generalizability; a comprehensive process evaluation guided by RE-AIM framework; our use of multiple evaluation methods including audio-recording of group sessions, on-site observations, and surveys as part of a comprehensive evaluation. Other strengths include the training of program staff using a standardized ongoing process; and adaptations to program materials, captured at multiple time points, which allowed for tailoring to t participant and health department characteristics. Despite these strengths, there were some study limitations. At the organizational level, there were a small number of health departments which limits our ability to generalize to NC local health departments. By randomly selecting sites using an optimized probability sampling protocol [24], we were able to meaningfully address this limitation. For intervention delivery monitoring, site visits were scheduled with each interventionist in advance; therefore, interventionists may have been more adherent to intervention protocols when directly observed because site visits were planned.

Conclusions
The outcomes of this process evaluation may serve as a guide on how to balance adaptations to t local settings while maintaining delity to core components. Still, there is more to learn regarding health departments' capacity to deliver and maintain such programs. The capacity of health departments to offer weight loss programs and services like the Weight-Wise II Program may re ect modi cations to budgets, sta ng, or programmatic priorities. Moreover, existing policies related to who can deliver reimbursable adult weight management services also impacts service delivery. Thus, identi cation of funding sources and the subsequent allocation to weight management programs and services warrant further investigation. Authors ContributionsAuthor contributions are as follows: acquisition of funding (CDSH); conception and design of the study (AFN, CDSH); data acquisition (AFN, CDSH); data analysis and interpretation (AFN, JC); drafting or substantively revising text (AFN, CDSH, LL, ASA, DSW, JC). All authors read and approved the nal manuscript.   ISSubmissionTIDieRChecklistWordrev18June20.pdf