Measuring Treatment Fidelity: From Research to Implementation

Background: Treatment fidelity is commonly assessed in research trials because greater fidelity helps later replication in practice and is generally associated with better outcomes. However, implementations after initial research may not be as faithful to the original protocol, leading to unsuccessful replication. Methods: Fidelity methods of the Resources for Enhancing All Caregivers Health in the VA (REACH VA) dementia caregiving clinical implementation were examined with comparisons to those of the Resources for Enhancing Alzheimer’s Caregivers Health (REACH II) randomized clinical trial. The NIH treatment fidelity conceptual framework was used, which includes design, provider training, delivery, receipt, and enactment. To translate these concepts into practice, for each REACH VA session, core activities that measured delivery, receipt, and enactment were identified. These core activities were then examined for a sample of 293 caregivers of veterans with dementia who received the REACH intervention from June 2012 to September 2014. Results: Methods to assess fidelity were not as extensive for REACH VA, compared to REACH II. However, fidelity methods were present in all five framework aspects. Overall completion of core items for delivery and receipt/enactment was 92%. Completion of core items declined as sessions continued, with delivery declining from 97.2% of items completed for Session 1 to 85.7% for Session 4; receipt/enactment declined from 97.3% to 88.4%. Conclusions: Consistent attention to maintaining treatment fidelity may explain why caregiver outcomes are similar in both the original REACH II randomized clinical trial and in the REACH VA implementation. The process followed to assess REACH fidelity is applicable to other evidence-based interventions that have been or are going to be implemented into practice. Fidelity assessment need not be continually re-invented, nor must it be elaborate. Critical components of an intervention as developed and practiced can and should be used to assess fidelity seamlessly. The interplay between how an intervention fits into the organization, ongoing assessment of its practitioners, and its effect on those participating in it should be seamless. In this way, fidelity assessment is not an add-on or burdensome for busy providers, supervisors, and organizations.


Background
To assess treatment benefit accurately, it is necessary to determine whether the treatment is being delivered as intended (delivery), whether participants are hearing and understanding (receipt), and whether participants are acting on the intervention (enactment). 1 An important reason to assess treatment fidelity is so an effective research intervention can be replicated and implemented into clinical practice. 2 Because delivery, receipt, and enactment are important in clinical practice, attention to treatment fidelity is common in research protocols. However, in practice, protocol implementation may not be as rigorous as the original, causing unsuccessful replication. This failure may diminish the demonstrated outcomes. It is, therefore, important to assess treatment fidelity in an implementation and compare it to the fidelity assessment of the original research protocol.
There are many ways to conceptualize and measure treatment fidelity, although all include delivery (treatment is provided as intended), receipt (participants are hearing and understanding), and enactment (participants are acting on the intervention). [2][3][4][5] In addition, the National Institutes of Health's (NIH) Behavioral Change Consortium 2,4 expands the number of aspects of fidelity to five. This expanded framework adds treatment design and provider training as separate aspects. In this framework, treatment design ensures that the intervention is congruent with relevant theory. 4 Based on theory, the intervention components hypothesized to affect outcome are made explicit in the protocol and highlighted in training and supervision of interventionists. 2 The NIH framework of design, provider training, delivery, receipt, and enactment was developed to conceptualize treatment fidelity in public health trials to improve future implementation and replication. 2,4 Treatment design and provider training influence delivery, specifically adherence or whether the intervention is delivered as the designers intended, exposure/dose or whether the amount of intervention prescribed by the designers is delivered, and quality or how well the interventionist delivers the intervention. Treatment delivery, in turn, influences participant responsiveness to receive and enact. For example, complexity of the intervention may affect how closely the interventionist follows protocol and how well the participant responds. 5,6 REACH VA. Resources for Enhancing All Caregivers Health in the VA (REACH VA) 7,8 was based on the randomized clinical trial (RCT) Resources for Enhancing Alzheimer's Caregivers Health (REACH II). 9 REACH is a multi-component intervention to help caregivers manage patient behavioral concerns and their own stress. It is structured to specify activities to occur at each session to ensure that the five main caregiving risk areas are covered, including information on the disease and its course, safety for the person with dementia, caregiver health and emotional well-being, social support, and management of problem behaviors. It is targeted through a Risk Assessment that identifies the caregiver's specific problems, stresses, or behaviors within the risk areas. The intervention provides education, support, and skills training. Skills include problem solving focusing on practical, actionoriented behavioral strategies, mood management/cognitive reframing to think differently about situations that cannot be changed, and stress management techniques. The caregiver and an interventionist work together. REACH II, 9/01 to 9/04, was a national, multi-site RCT funded by the National Institute on Aging and the National Institute of Nursing Research. The REACH II intervention had 12 one-hour, in-home and telephone sessions and five telephone support groups during 6 months. REACH VA has been a national, multi-site clinical program in the Department of Veterans Affairs (VA) Veterans Health Administration (VHA) system since 2011. REACH VA, delivered by telephone, telehealth, or face-toface, includes 4 sessions during 2 to 3 months, with an option for further sessions, based on need.
Outcomes are similar for REACH II and REACH VA (decreases in caregiver burden, depression, stress, anxiety, and frustration, decrease in reported time on duty/time providing care per day, decrease in number of troubling dementia behaviors being managed and safety concerns). 7-9 Both REACH II 9 and REACH VA [7][8]

Methods
Fidelity methods for the REACH VA dementia caregiving implementation implemented in the VA health care system and the REACH II randomized clinical trial were identified and compared. The NIH treatment fidelity conceptual framework was used, which includes design, provider training, delivery, receipt, and enactment. Table 1 shows the NIH framework treatment fidelity strategies, using the strategies developed to assess the fidelity of the REACH I interventions that formed the basis for REACH II. 10 To translate these strategies into practice, for each REACH VA session, core activities that measured delivery, receipt, and enactment were identified. Delivery, receipt, and enactment treatment fidelity data were then examined for a sample of 293 caregivers of veterans with dementia who received the REACH VA intervention from June 2012 to September 2014. This sample was pulled for an economic analysis of Veteran healthcare costs after caregiver participation in REACH. There were 73 VA staff nationwide who served as interventionists for the sample. REACH VA Program Documentation Forms for each session were scored for the number of delivery, receipt, and enactment core activities that should occur as specified in the protocol as part of the intervention. Additional activities could occur but were not counted. For example, during Session 2, safety material is presented. This is a core activity that must happen and was counted as present or absent. However, addressing a safety alert is an optional activity that might or might not occur, dependent on whether an alert was identified; this activity was not counted. Percentage of activities completed was computed for each session and across all sessions. Dosage, represented by the number of sessions caregivers completed, was also calculated.

REACH Treatment Fidelity -Design.
Treatment design encompasses decisions that are made as the treatment is developed that can facilitate or hinder fidelity. One important principle of treatment design is that the intervention's components are clearly linked to the theoretical framework. 2,4 The REACH intervention's ability to reduce caregiving stress may best be understood through stress/health process theory. Caregivers experience stress if they perceive that the demands placed on them are greater than their resources and capacity to manage those demands. 11 The Lazarus and Launier stress model 12 expands this concept to focus on actions that caregivers can take to cope with stress. Effective coping depends in part on information and skills directed toward diminishing, tolerating, or mastering situational demands and cognitive and emotional responses. A major component of REACH is problem solving to manage caregiver concerns and patient concerns causing stress and burden. Caregivers are also taught strategies to manage stress and cognitive restructuring skills to reduce distress over behaviors and circumstances not amenable to change. 13 The relationship of the REACH intervention components to the stress health process model is shown in Figure 1.

REACH Treatment Fidelity -Provider Training.
Methods of enhancing fidelity through provider training include standardizing training, assessing skills acquisition, and developing strategies to help providers deal with diverse types of participants. Post training, care must be taken to prevent skills drift. Ideally, providers would be hired who are best suited to deliver the intervention. 2,4 Although there are exceptions, for most VA providers, delivering REACH VA is not their exclusive job.
Therefore, hiring decisions are not made based on suitability to deliver the intervention, as they would be for a research study such as REACH II. However, the clinical skills and empathy to develop rapport are part of the usual skill set of those who deliver REACH. For example, the 73 staff who delivered REACH VA for the assessed sample included 30 psychologists, 36 social workers, 3 nurses, and 4 other clinicians.
Although training for REACH II was longer and more rigorous, as is appropriate for a research intervention, the components were like REACH VA. Both REACH II and REACH VA had the challenge of making sure training was standardized across multiple sites. As is often the case for multi-site clinical trials, REACH II training was conducted centrally through a Coordinating Center, using slides and telephone and lasted 2 days. REACH VA training is also conducted centrally, through the aegis of VA's Employee Education System. Staff register for a 3-hour webinar with slides and live lecture by the REACH Program Coordinator who delivers each training. The training is scripted but also involves participant interaction through poll questions, whiteboards for participant responses, and practice of skills such as problem solving. Continuing education credits are available.
Certification for both programs included a knowledge test and roleplay with a mock caregiver. The interventionist is rated on behavioral markers of specific procedural techniques (e.g., use of forms), clinical skills (e.g., active listening), components that are intended (positive), and plausible confounding parts that should not occur (negative). Suggestions for working with diverse types of caregivers, such as those with low literacy, are included during training and in the Program Coach Manual. For REACH II, a national certification committee provided feedback. For REACH VA, the program coordinator provides one-on-one feedback during the roleplay. In addition, coordinator and prospective interventionist have a one-hour consultation call to discuss the program, where and how the program will be delivered at the interventionist's facility, and any factors that may help or hinder implementation.

REACH Treatment Fidelity -Delivery.
To ensure that treatment is delivered as developed, content and dose must be specified, 2,4 preferably through a treatment manual. There needs to be a mechanism to measure the provider's adherence to the protocol's content and dose.
Just as for REACH II, the REACH VA Coach Manual specifies each component that should be delivered during each session. The Manual includes an overview of the program, the risk assessment, and an array of where information to address each risk assessment question can be found in the Caregiver Notebook. Troubleshooting tips and all forms are also provided. Each session has a schedule with components to be addressed, descriptions of what should be covered, suggested times for each component, talking points, and scripts if needed.
For example, the description of the Signal Breath stress management technique includes six tasks/benchmarks to be completed.
[1] These include: 1) introduce technique; 2) teach caregiver to rate level of tension; 3) describe technique; 4) practice with caregiver; 5) identify facilitators and barriers to practice; and 6) encourage use of technique. There is a script that covers each task. One difference between the two programs is in the length of time allotted to these tasks, reflecting the difference between a 12-session research study and a 4-session clinical program. For REACH II, 30 minutes are allotted; for REACH VA, 10 minutes.
For documentation, checklists include components of each session as a check off. However, for Signal Breath above, each task was listed for REACH II with places for notes; but for REACH VA, although each step is specified in the protocol, there is only the option to indicate present or absent for the entire teaching of Signal Breath. REACH VA interventionists have multiple means to document program delivery. Scanned paper or electronic fillable forms can be placed in the veteran's or caregiver's medical record. An electronic progress note mirrors the documentation form and is available in the electronic medical record.
Core delivery, receipt, and enactment activities include assessing the caregiver, reviewing materials and skills, introducing information and skills, practicing skills, developing problem solving plans, and rating the success of plans and strategies tried. Across all sessions the percentage of delivery activities that occurred was 91.9% (Table 2). Activities completed ranged from 97.2% of possible core activities completed in Session 1 to 85.7% of completed activities in Session 4. Mean dosage for the caregivers was 3.7 ± 1.7 sessions, with a range of 1 to 11 sessions. Of the sample, 67.2% of caregivers completed four or more sessions.

REACH Treatment Fidelity -Receipt
Assessing treatment receipt involves determining if participants have understood the intervention and if they can use the information and skills. Assessment methods can include pre-post tests, ensuring materials match health literacy level, and using multiple methods of presenting material, such as repeating information, queries about the material, and role-play with coaching and feedback. 2,4 For delivery, activities that the interventionist delivered are marked on the documentation form, but for receipt there is neither direct observation of the interaction between interventionist and caregiver nor does the caregiver self-report to an outside observer. Instead, items marked by the interventionist are used to infer receipt, using the steps in the REACH protocol. As part of the intervention, participant and interventionist work though problem solving, cognitive reframing, and stress management exercises together to ensure receipt. For example, the caregiver practices the Signal Breath and stretching with the interventionist. For other stress management techniques there are forms to help the caregiver plan. Items that are reviewed by interventionist and caregiver in identifying pleasant events include the caregiver's list of activities to try and a plan to ensure that an activity occurs. The plan includes the name of the activity, when it will occur, where it will occur, and what is needed to make it occur.
To help the caregiver learn the problem-solving process the caregiver answers questions about the chosen problem using the ABC process (antecedents, behavior, consequences) to pinpoint exactly what the problem is, who is present, and what is happening before and after the problem occurs. The caregiver then articulates a goal for the problem solving, generates possible solutions, and decides which is most likely to succeed. Caregiver and interventionist develop a plan to implement the solution, brainstorming barriers and enablers. The number of strategies is limited to no more than three, so the caregiver will not be overwhelmed. For REACH II, this joint plan was written up by the interventionist and then given to the caregiver at the next session. As an aid to the process, the REACH VA Caregiver Notebook, which is written at fifth grade reading level, has strategies listed in each chapter. Caregiver and interventionist highlight or otherwise identify strategies the caregiver will try. Finally, at the end of each session the caregiver is asked to make a written commitment that indicates what was discussed during the session and the specific strategy or strategies that the caregiver will try before the next session. Active learning strategies are encouraged; the interventionist may ask the caregiver to role-play any difficult solutions, practice asking for help, or try different communication strategies.
The percentage of core activities completed for receipt ranged from 97.3% in Session 1 to 88.4% in Session 4, and across all sessions the percentage of receipt activities that occurred was 92.2%.

REACH Treatment Fidelity -Enactment
Assessment of enactment involves determining if the participant has tried to implement the intervention. 2,4 Assessment methods can include direct observation, self-report, or provider report.
The structure of the REACH intervention facilitates enactment. Each session builds on the previous session as part of the REACH protocol, and the topics that were discussed during the last session are reviewed. Modifications are then made if needed. Commitments -the strategies the caregiver tried since the last session -are reviewed, and barriers to implementation are assessed. Solutions are discussed, and commitment strategies modified, if necessary. Goal attainment for each problem is based on a five-point rating scale, from "a lot worse" to "a lot better." If the problem is a lot better the caregiver has the option of choosing a different problem to work on next. If the problem is not a lot better, both the interventionist and caregiver review the last session's strategies and what the caregiver tried, discuss success and barriers and the caregiver's feedback and reflection, and select additional strategies to try. In REACH II caregivers had forms to complete between sessions to document enactment. For REACH VA it is not possible to separate receipt and enactment because the interventionist and caregiver work on strategies and review homework during the session. Therefore, their percentage of core activities completed for enactment is the same as for receipt. For example, reviewing a problem solving plan can be evidence of receipt and enactment, because the caregiver understood the material and was able to make a plan.

Discussion
We examined methods to assess treatment fidelity for REACH VA, one of the clinical translations of the REACH II RCT. The same or similar methods were used in the five aspects of treatment design, provider training, delivery, receipt, and enactment. Although data collection to assess fidelity was not as extensive in REACH VA as in REACH II, because it was a program being implemented by clinical staff, there is evidence that the intervention is being delivered, received, and enacted in the VA system. Completion of core items may have declined as sessions continued, but overall completion of items remained around 92%. Thus, attention to treatment fidelity may explain why caregiver outcomes are similar in the original study and in this clinical implementation.
Importance of treatment fidelity lies in the ability to correctly deliver interventions to real world patients. Once interventions have been replicated it is unclear how often treatment fidelity is assessed. However, ensuring that an evidence-based intervention remains true to the original protocol is important to maximize resources of the organization and the health and well-being of those being served. Thus, one way to plan for eventual clinical implementation is to build direct or indirect monitoring of treatment fidelity into a research study so that time and resource costs are minimal. 14,15 However, even with this advance planning, common methods to measure delivery, receipt, and enactment such as direct observation, video or audiotaping, and coding are labor intensive and may not be practical in a clinical setting. A balance must be struck when moving a program into practice by busy clinicians. 16,17 Methods to assess treatment fidelity must be in place to ensure that the program is delivered, received, and enacted as planned. If fidelity assessment methods are to be used, they need to be unobtrusive, feasible, and as much a part of business as usual as possible. The trade-offs between rigorous fidelity assessment and the customary practice of the organization and its personnel constitute a limitation of this REACH VA fidelity examination and, indeed, the examination of fidelity in any practical setting. For example, although the training, scripts, and session protocol specify activities that need to occur as part of each component, documentation for REACH VA only captures that the component did occur. The caregiver's receipt and enactment in the form of practice must be inferred. This is a limitation, and it is also likely to be the case in other clinical settings where documentation focuses on delivery methods under the clinician's control.
Documentation of the caregiver verbalizing understanding could be added as is done with other clinical documentation of patient/caregiver education.

Conclusions
The simplest approach to assess fidelity is that practices already part of the clinical environment can be dual-purposed. The certification process used in training staff to conduct the intervention can also be used as a competency checklist for performance appraisal. Treatment manuals and algorithms provide a listing of components and steps that should be followed. These can be mapped onto delivery, receipt, and enactment methods. Treatment documentation details their occurrence.
Clinicians and organization staff are accustomed to document for reimbursement and for quality assurance, and supervisors are accustomed to using documentation for supervision of staff and to evaluate the operation of a program. For example, the REACH VA template in the electronic medical record provides a means of documenting the conduct of the intervention. However, it is also used to capture staff workload for facility reimbursement.
Treatment fidelity measurement does not have to be continually re-invented nor must it be elaborate to link the conduct of the intervention and the measurement of fidelity. The same methods that were used for REACH II and REACH VA were first used to evaluate fidelity 10 in the six REACH I efficacy/feasibility trials, which tested different types of caregiving interventions (see The Gerontologist, 2003 volume 43, issue 4). Although outcomes, not fidelity, were the focus of these articles, critical aspects of fidelity assessment such as adherence, exposure/dose, delivery quality, and participant responsiveness, were similarly measured and reported as part of the interventions.
These same methods continue to be used to ensure fidelity in implementations of the REACH intervention, 16,17 including REACH VA.
The process followed to examine fidelity for REACH is applicable to other evidence-based interventions translated into practice. Critical components of an intervention can and should be the standards that are used to assess fidelity. Fidelity assessment cannot be an add-on or an additional burden for busy providers, supervisors, and organizations. The interplay between how an intervention fits into the organization, ongoing assessment of its practitioners, and its effect on those participating in it should be seamless. Successful translation of a clinical intervention can provide added value and improve the quality of care provided.