Implementation Fidelity of a Complex Behavioral Intervention to Prevent Diabetes Mellitus in Two Safety Net Patient-Centered Medical Homes in New York City

Abstract


Introduction
Implementation delity, de ned as the degree to which an intervention is implemented as planned, [1][2][3] is critical to inform our understanding of an intervention's outcomes [1,[3][4][5] by avoiding the so called Type III error where a useful intervention could be rejected for showing a lack of impact when in reality the lack of impact was caused by suboptimal implementation delity [1].In addition to clarifying causal mechanisms, assessment of implementation process can provide critical information to policy makers and evidence-based practitioners to guide future implementation of the intervention [1][2][3]6].Examining implementation delity is particularly necessary for multicomponent complex interventions because of their higher likelihood of deviation from the protocol [6].However, there are limited published studies of delity assessment of complex behavioral interventions [6,7].A review of psychosocial interventions found that only 3.5% of the interventions adequately addressed delity.[3] The CHORD trial (Community Health Outreach to Reduce Diabetes), launched in 2017, is a clusterrandomized pragmatic trial that is testing a Community Health Worker (CHW) driven intervention to promote healthy lifestyle changes and reduce the incidence of diabetes among a population of underserved patients with prediabetes in New York City safety-net hospital settings [8] There is evidence supporting the role of lifestyle interventions in preventing Type II diabetes mellitus (DM) among people with prediabetes and of the role of CHW's peer support in supporting positive behavioral and lifestyle transformations [9][10][11].As behavioral transformations are di cult to achieve, primary care (PC) visits are often insu cient and peer support, a strategy that focuses on training individuals from within the community, such as CHWs, to guide, educate, motivate, refer, track and support individuals to make and sustain behavior changes, can supplement PC visits to provide the required health coaching and social support for attaining behavior change goals [12].CHWs are trained members of the public health workforce who are trusted members of the community and are familiar with local resources available for the community members [8, 13,14].They serve to bridge the cultural and health literacy gaps between communities and health care systems by providing culturally congruent and personalized peer-coaching, leveraging their shared lived experience with patients [8, 13,14].CHW driven interventions have demonstrated promising results in improving health knowledge, behaviors and outcomes, in particular for underserved communities [13,15].The goal of the CHORD trial is to ll the gap in understanding how these strategies can be translated and implemented in the real world to foster positive behaviors in a population with prediabetes to prevent the onset of DM.
To balance the need for standardizing the intervention with responsivity to context [13,16], the CHORD intervention was developed with de ned core components and some optional components that allowed for varying intensities of implementation.This multicomponent, pragmatic, and community-based trial makes it a complex behavioral intervention with a high likelihood of deviation in the implementation of its protocol [17].The purpose of this study was to conduct quantitative concurrent assessment of the implementation delity of the core components of CHORD intervention and to examine factors affecting implementation delity using the Conceptual Framework for Implementation Fidelity.[1,5]

Description of the Intervention
The CHORD trial targets patients with prediabetes from PC clinics of the Manhattan campus of the VA NY Harbor Healthcare System (VA) and Bellevue Hospital Center (BH), of New York City's municipal hospital system [8].We described details of the protocol previously, [7] and will brie y summarize the core elements here.Eleven PC teams at the VA and four PC teams at BH were randomized within each site into intervention and control arms.Eligible patients were identi ed from PC panels if they were 18-75 years of age with at least one glycosylated hemoglobin (HbA1c) result in the prediabetic range (5.7-6.4%) in the 5 years prior to the start of the intervention, after excluding patients with a prior diabetes diagnosis.
The trial included patients who could speak English or Spanish to communicate with the bilingual CHWs.Each eligible patient in the intervention group was assigned to a CHWs based on language and neighborhood.Each CHW reached out to their assigned patients for a maximum of 10 times in attempt to enroll, after which the patient was dropped from further outreach.Once consented and enrolled, patients completed an intake survey.
The intervention included a 6-month intensive period followed by a 6-month maintenance period with monthly follow-up calls.The CHORD intervention was a complex intervention with four core components that were delivered by CHWs during in-person or telephonic encounters with patients.A minimum of 12 encounters, 6 in each of the two intervention periods was required.
As shown in the logic model in Fig. 1, the theory of change of the intervention was based on the delivery of the four core components -establishing personalized goals, educating, supporting PC visits and making appropriate referrals -by CHWs to affect behavioral lifestyle modi cations and prevent diabetes onset among patients with prediabetes.The rst intervention component required CHWs to establish individualized goals with each patient and complete a 6-item, Patient Activation Measure (PAM).[18]These goals were then translated into a health action plan (HAP), tailored to each patients' goals, PAM score, and preferences.Using the HAP, CHWs chose among 22 educational topics organized into 5 modules (i.e., prediabetes 101, healthful eating, MyPlate,[19] and plate portions, physical activity, and stages of change) to conduct education session(s) with patients, and provided an educational packet with information about diabetes, exercise and nutrition to prevent diabetes.Patients were encouraged to have at least one PC appointment during the intervention period and CHWs called patients before and/or after PC visits to encourage them to discuss diabetes prevention with the PC clinician or to review their after-visit summaries regarding diabetes.CHWs could also meet the patients at the clinic during their PC visit.Finally, CHWs discussed and facilitated referrals to hospital, community-based programs as needed.
In addition to these core activities, CHWs conducted regular check-in phone calls with their patients and could decide to offer other activities, such as home visits or group activities as appropriate.Postintervention, patients were contacted by a different staff member to complete a follow-up survey to assess their experience and satisfaction with the intervention, and changes they made in their diabetes prevention behaviors.To facilitate and standardize the implementation of the intervention, all CHWs received comprehensive training at the start and had ongoing feedback during weekly team meetings and case review sessions.These meetings allowed for troubleshooting of any issues and fostered exchange of experiences across CHWs and the research team, creating an environment of co-learning for best addressing concerns and challenges in implementation.
To manage CHW caseload limits, patient enrollment at the two implementation sites in the CHORD trial was staggered across four, overlapping waves.Each intervention arm patient was eligible to receive the 12-month intervention.As the COVID-19 pandemic hit New York City in March 2020, the implementation of the CHORD study was substantially impacted and required adaptation of trial procedures.Due to a lack of comparability of process indicators before and since the pandemic, this paper was restricted to the pre-pandemic CHORD trial waves, which includes the rst three waves at BH and two waves at VA. Data for these waves was collected over nearly 2 years from the start of the CHORD trial (December 2017) until October 2019.This study was approved by the NYU Langone Health (Protocol #:16-00690) and the VA (Protocol #:1609) Institutional Review Boards.

Guiding Framework
Data were collected in accordance with the modi ed version [5], of the Conceptual Framework for Implementation Fidelity (CFIF) originally proposed by Carroll et al, [1].The key indicator of implementation delity according to CFIF is adherence de ned as "whether a program service or intervention is being delivered as it was designed or written" [1].The domain of adherence is conceptualized as coverage ("what proportion of target group participated in the intervention") [5], content ("was each of the intervention components implemented as planned") [5], and dosage which includes duration and frequency ( "amount of an intervention received by participants" [1], or "were the intervention components implemented as often and for as long as planned") [5].In addition, the framework proposes to measure factors that moderate delity of implementation to investigate the reasons underlying any observed heterogeneity between different implementations of an intervention or deviations from the expected intervention which when combined with the adherence data, can provide a complete picture of delity.Context ("what factors at political, economical, organizational, and work group levels affected the implementation") [5] and participant responsiveness ("how were the participants engaged with the intervention services") were measured in this study to evaluate their effect on adherence.
Figure 2 shows the modi ed version of the CFIF along with the delity measures and moderators examined for the CHORD trial.

Data Collection
The data collection tools that were used to retrieve data for this study included standardized, Research Electronic Data Capture (REDCap) [20], forms completed by CHWs for each intervention patient to record their demographics, outreach and enrollment information, baseline intake information, and weekly logs about their encounters with each patient.Electronic Health Records from VA and BH provided other descriptive data on study patients and their healthcare utilization.

Measures
We report on the following four delity measures and two moderators.Table 1 describes the measures and data sources mapped to the framework constructs.a Percent of intake patients who established at least one goal or completed establishing a Health Action Plan was operationalized as a measure of two delity constructs -content adherence and coverage -because this component was the rst component that patients were required to complete in order to proceed with other components of the intervention b Duration of follow-up is the time from beginning of outreaching till the last successful encounter 1. Coverage was measured as the percent of outreached patients who were enrolled, percent of enrolled patients who completed intake, and percent of intake patients who continued with completing the rst core component of establishing at least one goal or a HAP, i.e. received some component of the CHORD intervention.
2. Content adherence of the four core components was measured as the percent of intake patients who established at least 1 goal or a HAP, received coaching on at least one education topic, received coaching on all education modules, had at least 1 PC visit, received at least one referral and who received all four core components in some capacity.In addition, we report the percentage of patients who had at least 1 successful encounter.
3. Dose -frequency of the four core components was measured as the median number of goals established and completed, median number of education sessions delivered and modules discussed, median number of PC visits, median number of referrals, and the median number of successful encounters.
Dose-duration was measured as the median duration of follow-up de ned as the time from beginning of outreach until the last successful encounter 4. Participant responsiveness was measured as the baseline Patient Activation Measure (PAM) score since activation has been associated with engagement with behavioral interventions.We categorized the PAM score as less than or equal to median vs. above the median.
5. Context was measured as the two clinical sites -BH and VA.

Analysis
Our primary analysis focused on delity measures and, as a secondary analysis, we assessed how these delity measures differed by (or were moderated by) PAM score and by clinical site.Fidelity measures of coverage, content adherence and dose were reported using applicable descriptive statistics which included percentage for categorical variables, or median with their inter-quartile ranges (IQRs) for continuous or count variables.To evaluate how delity measures were moderated by PAM score and clinical site, unadjusted p-value from Chi-square or Mann-Whitney U test and adjusted p-value from regression models that controlled for gender, language, race, ethnicity and age, were computed.For the regression model -the rst set of models treated PAM score as the primary independent variable, and the second set of models treated clinical site as the primary independent variable.Each set of models included separate models for each delity measure as the dependent variable.Logistic regression models were used for measures of coverage and content adherence which were modelled as binary variables (received or not received the respective component of the intervention).Linear regression models were used for delity measures of dosage that were continuous, and negative binomial regression models were computed to model the delity measure of dosage that was a count variable.
Among patients who completed intake (n = 444), the denominator for our delity assessment, 32 patients (7.2%) were missing the PAM score, and were excluded from the analysis examining moderation of delity measures by PAM score.Chi-square test or Mann-Whitney U test, as appropriate, were used to compare two population groups -how eligible/outreached patients differed by their enrollment status and how enrolled patients differed by their intake completion status.These comparisons were conducted for demographic characteristics including gender, language, race, ethnicity, age at outreach (years) and PAM score.A successful encounter between a patient and a CHW was de ned as an encounter (after the completion of the intake survey) where the CHW was able to speak with the patient either in-person or by phone, or when a letter or a text message was delivered (that is, when no evidence of failure to deliver was apparent).All quantitative analysis was conducted in the R statistical software environment, [21] and statistical signi cance was based on an alpha of 5%.

Results
Results presented below have been categorized into A) assessment of implementation delity measures (primary analyses) and B) the impact of moderators on delity measures (secondary analyses).

A) Fidelity Measures
Coverage A total of 1449 eligible patients were assigned to CHWs for outreach.After excluding 471 patients with incorrect/no contact information or who could not be reached after 10 phone calls, an additional 416 patients declined to participate or deferred and another 3 patients were found to be ineligible.This left 559 patients who enrolled in the intervention arm.Among enrolled patients, 444 (79.4%) completed an intake survey and were eligible to receive the intervention and comprised the analytic sample for the delity assessment (Fig. 3).Among the 1449 patients eligible for enrollment, the 559 enrolled and 890 unenrolled patients differed in their primary language, race, ethnicity, implementation site and median age at outreach (Table 2).However, among those enrolled, patients completing intake vs. those not completing intake were not signi cantly different on any of the measured characteristics (Table 3).Among intake patients, 362 received the rst core component of establishing a HAP.

Content adherence
Nearly 80% of the patients completing an intake established at least one goal for their HAP, received coaching on at least one education topic and had at least one PC visit, indicating high content adherence to these three core components of the intervention.In addition, 40% received coaching on all education modules.Less than half (45.0%) received a referral.Most of the referrals were for a healthcare visit, with the next most common reason for referrals being to facilitate access to healthy lifestyle, employment/workforce training, social security bene ts, or mental health services.Nearly 80% of patients had at least one successful encounter with their CHW.Overall, 34.9% patients received all four core components in some capacity (Table 4).b A successful encounter between a patient and a CHW was de ned as an encounter (after the completion of the intake survey) where the CHW was able to speak with the patient either in-person or by phone, or when a letter or a text message was delivered (that is, when no evidence of failure to deliver was apparent).The number of patients with a successful encounter is less than the number of patients who completed at least one goal or the number of patients who received coaching on at least one education topic because although not designed to, for some patients, after a patient completed an intake, goal establishment or education sessions happened without a successful encounter.

Dose -Frequency and Duration
Among the 362 patients who established a HAP, more than half established 3 goals (median 3; IQR: 2, 3), more than half completed at least 1 goal, and 25% completed 3 goals (median number of goals completed: 1; IQR: 0, 3).The median number of education sessions delivered was 18 (IQR: 8, 33) and the median number of education modules covered was 4 of 5 (IQR: 2, 4).Median number of referrals was 1 (IQR: 1, 3), with 25% receiving ≥ 3. The median number of PC visits was 3 (IQR: 2,5).A total of 4,072 encounters were documented, of which 55.6% (n = 2,155) were successful.The median number of successful encounters per patient was 5 (IQR: 3, 8) (Table 5).Among them, 40 (9.9%) had the intended minimum number of successful encounters which was twelve.The median duration of follow-up time was 411 days [IQR: 341, 446].Although the intervention was designed as a 6-month intensive period followed by a 6-month maintenance period, because of the real-world challenges of implementing a pragmatic community-based trial among vulnerable populations, the intervention phases did not adhere to the strict 6-month periods.Instead to accommodate variations in the responsiveness or needs of some patients, intensive contacts for some patients continued beyond 6 months and the overall intervention period extended over 12 months for some patients.

Patient activation
The median PAM score was 18.0 of a maximal score of 24.As shown in Table 6, none of the delity measures was moderated by PAM score above versus below the median.Two coverage measures -Percent of outreached patients who were enrolled and Percent of enrolled patients who completed an intake -were not evaluated for moderation because they do not make up our denominator (number of patients completing intake).These measures have been reported in text and Tables 3 and 4.
d Percent of intake patients who established at least one goal or completed establishing a Health Action Plan was operationalized as a measure of two delity constructs -content adherence and coverage -because this component was the rst component that patients were required to complete in order to proceed with other components of the intervention e Calculated among subjects who established at least one goal

Context
We found that the implementation of the CHORD trial was moderated by clinical site, a measure of context in our study.Nearly 60% in this analytic cohort were from BH and 40% from the VA.We found coverage and content adherence to be higher at VA than at BH, except for the percent of patients who received coaching on all education modules and the percent of patients who received all four components in some capacity, which was higher at BH than at VA. Three content adherence measures, percent of patients who received at least one referral, at least one successful encounter and at least one PC visit were not different between the two sites.Three dose-frequency measures, number of PC visits (median value: 4 VA vs. 4 BH), number of education modules covered(median value: 4 VA vs. 4 BH), and the number of successful encounters (median value: 4.0 VA vs. 6.0 BH), differed between the two sites.
(Table 7).Two coverage measures -Percent of outreached patients who were enrolled and Percent of enrolled patients who completed an intake -were not evaluated for moderation because they do not make up our denominator (number of patients completing intake).These measures have been reported in text and Tables 3 and 4.
c Percent of intake patients who established at least one goal or completed establishing a Health Action Plan was operationalized as a measure of two delity constructs -content adherence and coverage -because this component was the rst component that patients were required to complete in order to proceed with other components of the intervention Our analysis demonstrated moderate to high rates of implementation delity in this trial.We found that CHWs were able to complete an intake with nearly 80% of the patients enrolled in the intervention arm, and three of the four core components (goal setting, PC visit and education) were delivered to nearly 80% of the patients.While the component of facilitating a referral was delivered to only 45% of the patients, a quarter received 3 or more referrals.While coverage and content adherence were moderate to high for the four components, we found high variability in the dosage of these core components.As compared to a minimum of 6 required encounters per patient in the intensive and maintenance phase respectively, the median number of successful encounters in the intensive and maintenance phases in total was only 5(IQR: 3,8).In addition, nearly 66% completed the intensive phase of the intervention, with an average follow-up time of more than a year.Given the planned duration of intervention, which was 1 year, the observed average follow-up time is a measure of implementation success.
In a complex intervention, there is potential for deviations from the planned program.[1] Lifestyle interventions, such as the one being tested by the CHORD trial, are inherently complex because they involve multiple dimensions, have diverse interacting components and target several organizational levels.This makes their success highly dependent on a multitude of real-world variables.[1,2] These real world variables can precipitate deviation of the 'intervention as designed' from the 'intervention as delivered'.[22] Using multiple regression models we assessed the delity of implementation by two among the ve moderating factors described in the modi ed CFIF, [1,5] a contextual factor (the site of implementation, BH vs. VA) and a measure of patient activation (PAM score).While the implementation site moderated several measures of content adherence and dosage, patient activation was not associated with delity.
Our study used PAM score to measure participant responsiveness.In light of evidence which shows that patient activation-"the skills and con dence that equip patients to become actively engaged in their health care" contributes to positive health outcomes, [23] null ndings in our process evaluation were unexpected.However, these ndings might be explained by emerging evidence indicating that interventions tailored to a patient's level of activation can build skills and con dence, thereby increasing patient activation.[23] The components of the CHORD trial were tailored to the patient activation measure at the time of intake, which might have affected their activation and engagement throughout the intervention.Therefore, a comparison of the quality of implementation by baseline PAM score might not have revealed any differences.It is also important to note that PAM score at intake might not be directly associated with the perception and view of patients about the relevance of an intervention, which has been suggested to directly moderate implementation delity by impacting the engagement of patients.[1] Hence, PAM score at intake might be limited in its representation of the concept of 'participant responsiveness' as de ned in the CFIF, and of the concept echoed in Rogers' diffusion of innovations Fidelity assessment is key to understanding the reasons for the success or failure of interventions.
However, very few studies systematically document and report implementation processes of their intervention programs.[3,5] The CHORD protocol built in collection of data regarding several delity measures and moderating factors that can impact delity.This allowed us to conduct concurrent rather than retrospective process evaluation.Concurrent process evaluations are important, as they can capture implementation experiences in real time and ensure that the theory behind the intervention is accounted for during evaluation.
[6] In addition to reporting on concurrent process evaluation of a pragmatic complex trial, the study responds to the general calls for conducting quantitative evaluations of delity in intervention studies.Use of real time data reported by the key implementors, the CHWs adds to the validity of our analysis.
Another major contribution of our study is in empirically testing the CFIF.The framework provided a useful tool for conceptualizing and organizing measures of delity and their moderators.These ndings, when evaluated with the data on CHORD's outcomes, can help address questions about the relative importance of the measures of implementation delity as described in the framework.These evaluations can also provide information on the predictive validity of delity measures.Our process evaluation also highlights the limitations of the CFIF for selecting measures of potential moderators.To standardize quantitative delity assessments, the eld will bene t from further guidance about "how-to" measure and apply factors that can moderate implementation delity.

Conclusion
Our concurrent quantitative process evaluation of a complex pragmatic trial, using the Conceptual Framework for Implementation Fidelity, found moderate-to-high adherence to the core components of the trial, as well as moderation of several delity measures by implementation site.This study adds to the growing knowledge base on the evaluation of implementation delity of trials, which is important to understand "how" and "why" an intervention fails or succeeds.For complex behavioral interventions, such as the one evaluated in this study, delity assessment becomes further critical because of the high potential of deviation from the planned protocol.This empirical test of the framework found it to be a useful tool for organizing and measuring delity measures as well as their moderators.However, our process evaluation also highlights the current lack of guidance on quantitative indicators of the proposed moderators.

Declarations
Ethics approval and consent to participate: This study was approved by the NYU Langone Health (Protocol #:16-00690) and the VA (Protocol #:1609) Institutional Review Boards.
Consent for publication: Not applicable

d
Calculated among subjects who established at least one goal e p values compare rank sums which are signi cantly different but medians are equalDiscussionThe theory of change of the CHORD intervention is that active, culturally congruent engagement of patients with prediabetes with trained CHWs, through individualized goal setting, educational coaching, supplementation of PC visits and facilitated referrals, will support positive lifestyle changes and prevent the onset of diabetes.However, interventions will unlikely affect lifestyle change if they are not implemented with delity.This study conducted a concurrent process evaluation of the core components of the CHORD trial intervention, to examine the extent to which the study team successfully implemented each of the core components of the program (implementation delity).The study provided additional information for implementation efforts by examining the factors that in uenced the extent of implementation ( delity moderators).

Figures Figure 1 Logic
Figures

Table 1
Study Measures and Their Data Sources Mapped to the Framework Constructs a Percent of intake patients who established at least one goal or completed establishing a Health Action Plan was operationalized as a measure of two delity constructs -content adherence and coverage -because this component was the rst component that patients were required to complete in order to proceed with other components of the intervention b Duration of follow-up is the time from beginning of outreaching till the last successful encounter

Table 2
Characteristics of Patients Who Were Determined Eligible for the Intervention Arm (and were Outreached by CHWs), by Their Enrollment Status

Table 3
Characteristics of Patients Who Were Enrolled in the Intervention Arm, by their Intake

Table 4
a The number of patients who received coaching on at least one education topic were more than the number of patients who established at least one goal (or a Health Action Plan) because although not designed to, 5 patients received an educational coaching session before they established a goal (or a Health Action Plan).

Table 5
a Duration of follow-up is the time from beginning of outreaching till the last successful encounter

Table 6
Moderation of Fidelity Measures by PAM Score Among Intervention Patients Completing the Intake Survey a Among patients who completed intake, 32 patients didn't have a PAM score recorded b P values are adjusted for gender, language, race, ethnicity and age through multivariate regressions c

Table 7
Moderation of Fidelity Measures by Clinical Site* Among Intervention Patients Completing the Intake Survey a P values are adjusted for gender, language, race, ethnicity and age through multivariate regressions b