Sustainment of Diverse Evidence-Based Practices Disseminated in the Veterans Health Administration (VHA): Development and Administration of a Pragmatic Instrument

doi:10.21203/rs.3.rs-951440/v1

Background

There are challenges associated with measuring sustainment of evidence-based practices (EBPs). First, the terms sustainability and sustainment are often falsely conflated: sustainability assesses the likelihood of an EBP being in use in the future while sustainment assesses the extent to which an EBP is (or is not) in use. Second, grant funding often ends before sustainment can be assessed.

The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) is one of few large-scale models of diffusion; it seeks to identify and disseminate practices across the VHA system. The DoE sponsors “Shark Tank” competitions, in which leaders bid on the opportunity to implement a practice with 6 months of implementation support. As part of an ongoing evaluation of the DoE, we sought to develop and administer a pragmatic instrument to assess sustainment of DoE practices.

Methods

In June 2020, surveys were sent to 64 facilities that were part of the DoE evaluation. We began analysis by comparing alignment of quantitative and qualitative responses; some facility representatives reported in the open text box of the survey that their practice was on a temporary hold due to COVID-19 but answered the primary outcome question differently. As a result, the team reclassified the primary outcome of these facilities to Sustained: Temporary COVID-Hold. Following this reclassification, the number and percent of facilities in each category was calculated. We used directed content analysis, guided by the Consolidated Framework for Implementation Research (CFIR), to analyze open text box responses.

Results

A representative from forty-one facilities (64%) completed the survey. Among responding facilities, 29/41 sustained their practice, 1/41 partially sustained their practice, 8/41 had not sustained their practice, and 3/41 had never implemented their practice. Sustainment rates increased between Cohorts 1 – 4.

Conclusions

The development and administration of our pragmatic survey allowed us to assess sustainment of DoE practices. Planned updates to the survey will enable flexibility in assessing sustainment and its determinants at any phase after adoption. This assessment approach can flex with the longitudinal and dynamic nature of sustainment, including capturing nuances in outcomes when practices are on a temporary hold.

Health Law

Health Economics & Outcomes Research

Sustainability

Sustainment

Measurement

Outcomes

Model of Diffusion

Consolidated Framework for Implementation Research (CFIR)

The terms sustainability and sustainment are used interchangeably in the literature; this paper provides clarity in defining and differentiating these terms.
Sustainment determinants and outcomes are often conflated in the literature; this paper illustrates that many sustainment determinants are inaccurately described as outcomes.
Sustainment is dynamic; this paper provides an approach to better capture nuance in sustainment outcomes when practices are on a temporary hold.
A high rate of practice sustainment among responding facilities suggests that the VHA DoE is a promising large-scale model of diffusion.

Evaluating Sustainment of Evidence-Based Practices is Challenging

There is growing interest in sustainment of evidence-based practices (EBPs); however, the literature on how to best measure sustainment over time is still developing. Understanding sustainment of EBPs is challenging, which Birken et al. suggest is due to a lack of conceptual clarity and methodological challenges [1].

First, the terms sustainability and sustainment are often used interchangeably [1]. While these terms are related, there are important distinctions. Sustainability assesses the likelihood of an EBP being in use at a future point in time; it is measured by assessing contextual determinants (“factors which decisively affect the nature or outcome of something”) [2]. For example, the EBP is perceived to have low sustainability due to inadequate funding or lack of priority. Operationally, the goal is to determine whether the conditions indicative of sustaining EBPs are in place, and if not, to guide efforts to put such conditions into place [3,4].

In contrast, sustainment assesses the extent to which an EBP is (or is not) in use after a specific period of time after initial implementation; for example, the RE-AIM Framework specifies that the sustainment period begins at least 6 months after initial implementation is completed [5]. Sustainment is measured by assessing outcomes (“the way a thing turns out; a consequence”) [6], e.g., the EBP is in use/not in use. Operationally, the goal is to determine if EBPs are still in place following the end of implementation support [7]. Distinguishing between sustainability and sustainment will help researchers develop shared language and advance implementation science [1,8].

Second, grant funding periods often end after implementation is completed, so initial and long-term sustainment cannot be assessed due to time and resource constraints [1]. As a result, most measure development has focused on sustainability (which can be measured at any point in time during grant funding periods) not sustainment (which cannot be assessed until after a sufficient amount of time has elapsed) [9].

Limitations to Current Sustainment Instruments

The existing literature conceptualizes a mix of items as sustainment outcomes, including the presence or absence of an EBP after implementation is completed, such as the continued use of the EBP (and its core components) [9–12] or the level of institutionalization of the EBP [11,12]. In addition, the literature discusses continued “attention to the issue or problem” addressed by the EBP, even when the specific EBP is no longer in use or is replaced by something else, as a sustainment outcome [11]. Finally, there are several outcomes referenced in the literature that have been used to measure both sustainability and sustainment, such as continued institutional support [9–12] and continued funding for the EBP [9], as well as the continued benefit of the EBP [10–12] (see Table 1). In effect, there is overlap in the literature between sustainment determinants and sustainment outcomes.

Although existing literature offers a variety of single-item sustainment measures for researchers to use, there are few complete pragmatic multi-item instruments. A narrative review by Moullin et al. identified 13 instruments for measuring sustainment. However, they highlighted the need for more pragmatic approaches since many of the existing multi-item sustainment instruments were “overly intervention or context specific” and “lengthy and/or complex” [13]. Furthermore, most multi-item instruments were not well-suited for frontline employees to complete; they were more suited for individuals with expertise in implementation science frameworks [13]. Pragmatic instruments are needed to increase the likelihood participants will understand and respond to all items, especially when it is difficult to incentivize participants over time.

Veterans Health Administration: Diffusion of Excellence

The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) is one of few large-scale models of diffusion; it seeks to identify and disseminate EBPs across the VHA system. DoE practices include innovations supported by evidence from research studies and administrative or clinical experience [14,15], and strive to address patient, employee, and/or facility needs. The DoE sponsors “Shark Tank” competitions, in which regional and facility leaders bid on the opportunity to implement a practice with 6 months of external implementation support. For additional detail on the DoE, see previous publications [7,16–18]. Over 1500 practices were submitted for consideration between Cohorts 1 – 4 of Shark Tank; the DoE designated 45 as Promising Practices and these were adopted at 64 facilities (some practices were adopted by more than one facility). Two additional practices were designated as Promising Practices but were implemented outside of standard DoE processes; these are not included in this evaluation. See Additional File 1 for practice descriptions.

In earlier phases of our evaluation, we focused on implementation and initial sustainment of DoE practices [18]. In brief, we conducted interviews after the 6-month external implementation support period to understand the level of implementation success as well as barriers and facilitators to implementation at the facilities [18]. Participants described a high level of successful implementation after the initial 6-month period of support. Due to extensive external implementation support, facilities were able to complete implementation unless significant barriers related to “centralized decision making, staffing, or resources” delayed implementation [18]. We then evaluated the initial sustainment of the practices asking facilities to complete follow-up surveys (on average 1.5 years after external support ended). Over 70% of the initially successful teams reported their practice was still being used at their facility. Additionally, over 50% of the initially unsuccessful teams reported they had since completed implementation and their practice was still being used at their facility [18]. Although some of these initially unsuccessful facilities implemented their practice after external support ended, research suggests that many EBPs are not sustained once implementation support has ceased [19]. As a result, we shifted our focus to the evaluation of ongoing sustainment of DoE practices.

As part of an ongoing evaluation of the VHA DoE and to address limitations in the sustainment literature, the objective of this manuscript is to: 1) describe the development of a pragmatic sustainment instrument and 2) present results on ongoing practice sustainment, as well as barriers and facilitators to sustainment for DoE practices.

Survey Development

To assess the ongoing sustainment of DoE practices, we sought to develop a pragmatic survey that was: 1) easy to understand for those without implementation science expertise (i.e., simple), 2) quick to complete (i.e., less than 10 minutes), and 3) appropriate for 45 different practices (i.e., generic) [13,20]. Our primary evaluation question for the survey was: Is there ongoing sustainment of DoE practices? To assess this question, we used the last known status of a facility (based on the last interview or survey completed) and branching logic to route respondents through the survey based on their individual facility’s situation (see Figure 1). If the last known status of a facility’s practice was “not implemented” (the facility had not completed implementation) or not sustained (the facility had completed implementation but had since discontinued the practice), the introductory survey question asked if the facility had completed implementation or re-implemented the practice. If the answer was “No”, they were asked about intentions to do so in the future, and then routed to the end of the survey. If the answer was “Yes”, they were then routed to the beginning of the sustainment survey. Based on our working definition of sustainment, items were conceptualized as primary or secondary outcomes; secondary items were derived from the literature to enhance the survey and provide additional contextual information (see below and Table 1). Furthermore, “Please Describe” open text boxes were included following all questions so participants could provide additional detail. Descriptions of each outcome are briefly described below; see Table 1 for outcomes mapped to the literature and Additional File 2 for the complete survey.

Terms and Definitions

Primary Outcome

As described earlier, our primary outcome is used as the overarching benchmark to determine if a DoE practice is sustained.

Practice Sustainment

Extent to which the DoE practice and its core components and activities are in use [9–12]

Secondary Outcomes

Given the importance of assessing more than whether the practice was in use, the survey included several items from the literature as secondary outcomes. These secondary outcomes provide additional information on the current status of the practice.

Practice Institutionalization	Extent to which the DoE practice is part of routine care and work processes [11,12]
Practice Priority	Extent to which there is attention to the issue or problem addressed by the DoE practice, i.e., “heightened issue salience” [11]
Practice Buy-in/Capacity/Partnership	Extent to which key stakeholders and partners support the DoE practice [9–12]
Practice Funding	Extent to which funding is provided to support the DoE practice [9]
Practice Benefit	Extent to which the DoE practice is having the intended outcomes [10–12]
Practice Improvements/Adaptation	Extent to which the DoE practice is being improved and/or adapted [10]
Practice Spread/Diffusion	Extent to which the DoE practice is spreading or diffusing to other locations [11]

Additional Survey Questions

To assess the fluid and longitudinal nature of sustainment, if a respondent answered “No” to the primary outcome, i.e., the DoE practice was not in use (see above and Table 1) they were asked about future plans to re-implement. This is similar to the question given at the beginning of the survey to facilities that had a last known status of no sustainment (see Survey Development above). If a facility’s representative reported they planned to re-implement their DoE practice, they were retained in the sample for future sustainment surveys.

Due to the timing of the sustainment survey (only a few months after the Centers for Disease Control and Prevention (CDC) issued guidance to cancel and/or reschedule non-essential clinical activities) [21], it included questions about the COVID-19 pandemic. These questions assessed the impact the pandemic had on their practice as well as how participation in the DoE impacted their response to the pandemic (see additional File 2).

Table 1

Survey Questions Mapped to Published Sustainment Outcomes

Sustainment Items	Survey Question	Palinkas [9]	Lennox [10]	Scheirer Dearing [11]	Shelton [12]
Primary Outcomes
1. Practice Sustainment	Is this practice still being used or done at your site? (Yes/No/Partially)	X	X	X	X
Secondary Outcomes
2. Practice Institutionalization	Is this practice considered routine, usual practice? (i.e., practice is nearly always used or done when appropriate by all individuals involved) (Yes/No/Partially)			X	X
3. Practice Priority	This practice has priority at your site. (Strongly Disagree to Strongly Agree Likert Scale)			X*
4. Practice Buy-in/ Capacity/ Partnership^†	This practice has support and commitment from facility leadership. This practice has a Champion (leader) at your site. This practice has sufficient staffing. This practice has support and buy-in from key outside community entities. (Strongly Disagree to Strongly Agree Likert Scale)	X	X	X	X
5. Practice Funding^†	This practice has sufficient funding. This practice has sufficient resources (e.g., space, equipment). (Strongly Disagree to Strongly Agree Likert Scale)	X
6. Practice Benefit^†	Is this practice demonstrating effectiveness at your site? (Yes/No/Partially)		X	X	X
7. Practice Improvements/ Adaptation	Have there been any changes or adaptations to this practice? (Yes/No/Partially)		X
8. Practice Spread/ Diffusion	Has this practice spread to other units or places in your site? (Yes/No/Partially)			X
* Scheirer & Dearing conceptualized this measure as issue priority, not practice priority; however, given that DoE practices assessed different issues, this was presented as practice priority [11]. † A similar item was conceptualized as a sustainment determinant by one or more authors represented in the table [9–12]

Data Collection

In June 2020, surveys were emailed by a member of the evaluation team to representatives of the 64 facilities in Cohorts 1 – 4 that adopted one of the 45 DoE Promising Practices and received 6 months of external implementation support. See Additional File 1 for practice descriptions. Survey follow-up periods ranged from 1 to 3 years, depending on cohort (i.e., when the practice was adopted). Incentives were not provided to VHA employees because surveys were expected to be completed during working hours. The survey was administered using the REDCap® platform. Per regulations outlined in VHA Program Guide 1200.21, this evaluation has been designated a non-research quality improvement activity.

Data Analysis

We calculated the overall response rate and used descriptive statistics (number, percent) to summarize the multiple choice and Likert scale questions. We used directed content analysis, guided by the Consolidated Framework for Implementation Research (CFIR), to analyze open text box responses [22]. The CFIR is a determinant framework that defines constructs across five domains of potential influences on implementation: 1) Characteristics of the Intervention (e.g., Evidence Strength & Quality), 2) Outer Setting (e.g., Patient Needs & Resources), 3) Inner Setting (e.g., Tension for Change), 4) Characteristics of Individuals (e.g., Self-Efficacy), and 5) Process (e.g., Planning). The codebook included deductive CFIR constructs as well as new inductive codes and domains that arose in the data, including relationships between constructs [23]. We used relationship coding to provide a high-level overview of how different constructs interact or relate to each other. In some cases, the relationship was unknown or not applicable, however, in other cases, it was helpful to describe the relationships between codes. See Table 2 for an excerpt of our CFIR informed codebook. Using a consensus-based process [23], two researchers (CR, AN) coded qualitative data from the open text boxes and discussed to resolve discrepancies.

Table 2

Codebook for Open Text responses to Survey Questions

CFIR Construct Codes	Operationalized Definitions
Characteristics of the Innovation Domain	Codes capturing information specific to the practice regardless of where it is being implemented
*Innovation Type
• Essential vs. Non-Essential	A practice that provided essential care vs. non-essential care in the context of the COVID-19 pandemic
• Virtual vs. In-Person	A practice that was virtual vs. in-person
Outer Setting Domain	Codes capturing information specific to the setting outside the facility
*Community Characteristics	Stakeholder perception of community characteristics impacting the practice, including but not limited to socio-cultural (e.g., white-supremacy, ableism), socio-economic (e.g., social assistance, housing), socio-political (e.g., government), and/or socio-geographical (e.g., built environment) characteristics.
Patient Needs & Resources	Stakeholder perception of patient needs, preferences, and resources
External Policies & Incentives	Stakeholder perception of policies impacting the practice, e.g., the CDC’s guidance regarding the COVID-19 pandemic, VHA policy changes
Inner Setting Domain	Codes capturing information specific to the setting within the facility
*Employee Needs and Resources	Stakeholder perception of employee needs, preferences, and resources
Tension for Change	Stakeholder perception of the level of need for the practice, including relevance or irrelevance of the practice during the COVID-19 pandemic
Compatibility	Stakeholder perception of the compatibility of the practice with existing workflows and processes
Available Resources	Stakeholder perception of the resources available to support the practice, e.g., space, equipment
Process Domain	Codes capturing information specific to the implementation and/or sustainment process
Executing
• *Adapting	Stakeholder perception of the extent to which the practice was or was not adapted
Engaging
• *Key Stakeholders	Stakeholder perception of the extent to which employees are available and/or engaged to deliver or do the practice
*Outcomes Domain	Codes capturing primary outcomes
Sustained: Ongoing	The practice is in use and ongoing
Sustained: COVID-Hold	The practice is in use but on a temporary hold due to the COVID-19 pandemic
Partially Sustained	The practice is partially in use, e.g., only some components of the practice are in place
Never Implemented/Not Re-Implemented/Not Sustained	The practice is not in use
*Relationship Codes	Codes (denoted by symbols) capturing relationship between constructs
\|	Hindered and/or stopped
>	Facilitated and/or led to
* A new inductively derived construct or domain

Primary Outcome

We began analysis by comparing alignment of quantitative and qualitative responses to the primary outcome (i.e., “Is this practice still being used or done at your site?”) (see Table 1: Item 1). Seven facility representatives reported in the survey’s open text box that their practice was on a temporary hold due to COVID-19 but answered the primary outcome question differently; two answered “Yes”, two answered “No”, and three answered “Partially”. As a result, the team reclassified the primary outcome of those facilities into a new category under Sustained: Temporary COVID-Hold (see Figure 2). Following this reclassification, the number and percent of facilities in each sustainment category was calculated by cohort.

Secondary Outcomes

We calculated the number and percent of facilities for Items 2 – 8 (see Table 1) within each of our primary outcome categories from Item 1 (Sustained, Partially Sustained, Not Sustained) (see Table 1: Item 1). We also analyzed the concordance between Items 1 (Practice Sustainment) and 2 (Practice Institutionalization) in the survey.

Primary Outcome

A representative from forty-one facilities (41/64; 64%) completed the survey in summer 2020 while 23 (35.9%) facility representatives were lost to follow-up; the rate of missing data was lower after the first DoE cohort. Among responding facilities, 29/41 (70.7%) facilities were sustaining their practice, 1/41 (2.4%) facilities were partially sustaining their practice, 8/41 (9.5%) facilities were not sustaining their practice, and 3/41 (7.3%) facilities had never implemented their practice (see Table 3). Sustainment rates increased across Cohorts 1 – 4. The CFIR constructs and inductive codes associated with primary outcome text responses are included in parentheses below; the facilitates/leads to relationship is illustrated with “>” and the hinders/stops relationship is illustrated with “|”. Please refer to Table 2 for code definitions.

Table 3

Practice Sustainment by Cohort: Number (Percent) of Facilities

Cohort (Year^*)	Sustained^†	Partially Sustained	Not Sustained	Never Implemented	Total Non-Missing	Missing	Total
1 (2016)	4 (50.0)	0 (0.0)	3 (37.5)	1 (12.5)	8 (47.1)	9 (52.9)	17
2 (2017)	8 (66.7)	0 (0.0)	3 (25.0)	1 (8.3)	12 (71.0)	5 (29.4)	17
3 (2018)	7 (70.0)	1 (10.0)	2 (20.0)	0 (0.0)	10 (71.4)	4 (28.6)	14
4 (2019)	10 (90.9)	0 (0.0)	0 (0.0)	1 (9.1)	11 (68.8)	5 (31.3)	16
Total	29 (70.7)	1 (2.4)	8 (19.5)	3 (7.3)	41 (64.1)	23 (35.9)	64
* Indicates year in which facilitated implementation support from DoE ended † This category includes 7 facilities that were on a temporary hold due to the COVID-19 pandemic.

Sustaining Facilities

Twenty-nine facilities (N = 41, 70.7%) were sustaining their practice (see Table 3). Of these 29 facilities, 22 (75.9%) were ongoing during the COVID-19 pandemic while 7 (24.1%) were on a temporary COVID-Hold (see Table 4). The differences between these two sustaining groups of facilities are described below.

Sustaining Facilities: Ongoing

In late March 2020, the Centers for Disease Control and Prevention (CDC) issued guidance to cancel and/or reschedule “non-essential clinical activities, including elective procedures, face-to-face outpatient visits, diagnostic testing, and procedures” [21]. However, 22/29 facility representatives (75.9% of sustaining facilities) reported their practice was ongoing during this time. Many clinical practices were able to continue because they provided essential care for patients or were already virtual in nature (Innovation Type: Essential or Virtual & Tension for Change > Sustained: Ongoing). In fact, the pandemic served to increase the need and therefore the spread of virtual practices:

“[Virtual care] is under a huge expansion. We are just now looking at adding Nursing [virtual care] clinics […] everything [virtual care] has expanded with COVID.” (Facility 4_IF02a)

In contrast, other practices were ongoing during the pandemic because they adapted the practice’s in-person events to virtual events (External Policies & Incentives > Adapting > Sustained: Ongoing):

“We are currently orchestrating our third annual Summit (virtually because of COVID).” (Facility 3_IF09c)

The other ongoing practices were designed to benefit employees or represented administrative process changes that were not impacted by the pandemic (Employee Needs and Resources > Tension for Change > Sustained: Ongoing):

"As a [department] we use this regularly and inform our employees of their current status as we continue to perform our normal tasks and duties.” (Facility 2_IF06a)

Table 4

Sustained Practices: Ongoing vs. COVID-Hold: Number (Percent) of Facilities by Cohort

Cohort (Year^*)	Ongoing	COVID-Hold	Total Sustained
1 (2016)	3 (75.0)	1 (25.0)	4
2 (2017)	5 (62.5)	3 (37.5)	8
3 (2018)	7 (100.0)	0 (0.0)	7
4 (2019)	7 (70.0)	3 (30.0)	10
Total	22 (75.9)	7 (24.1)	29
* Indicates year in which facilitated implementation support from DoE ended

Sustaining Facilities: COVID-Hold

Although the majority of sustaining facilities were ongoing, 7/29 facility representatives (24.1% of sustaining facilities) reported they placed their practice on a temporary hold following the CDC guidance [21] (External Policies & Incentives & Innovation Type: Non-Essential > Sustained: COVID-Hold). As illustrated in the following quotes, these facilities could not reasonably nor safely adapt their practice and offer it virtually (Patient Needs & Resources | Adapting > Sustained: COVID-Hold):

“Due to COVID-19, we are unable to use this program at this time. We are currently being encouraged to do telehealth from home. We believe this program would carry additional risks [to Veterans] should it be used by telehealth rather than face to face.” (Facility 2_IF11_2)

Other practices became less applicable when very few patients were present in the hospital, e.g., practices seeking patient feedback or reporting patient metrics (External Policies & Incentives | Tension for Change > Sustained: COVID-Hold).

“Due to the pandemic we did not have the metrics to utilize the [practice] so it was placed on hold.” (Facility 1_IF05)

Partially Sustaining Facility

Only one facility (N = 41, 2.4%) was partially sustaining their practice (see Table 3). The respondent explained partial sustainment by noting the practice was in use “in some specialty clinics, palliative care and hospice.” (Facility 3_IF04)

Not Sustaining Facilities

Eight facilities (N = 41, 19.5%) were not sustaining their practice (see Table 3). Within this group, 6/8 (75%) had a previous last known status of no sustainment and 2/8 (25%) had a previous last known status of sustained or partially sustained.

Not Sustaining Facilities: Facilities that were previously not sustaining

As noted in the Methods section (see Survey Development), facilities that had a last known status of no sustainment were given an introductory question to determine if they had re-implemented their practice in the interim. Six facilities (N = 8, 75% of the not sustaining facilities) had not re-implemented for various reasons. Two of these facilities had not re-implemented due to losing necessary staffing and not having completed re-hiring (Engaging Key Stakeholders > Not Re-Implemented).

“[The] person that initiated this practice left and it was not followed through with new staff.” (Facility 2_IF07b)

Two other facilities had not re-implemented because the practice was incompatible with patient needs, facility resources, or existing workflows (Patient Needs & Resources & Available Resources | Compatibility > Not Re-Implemented).

“[The practice] did not meet the needs of our Veterans in [service] [and there were] issues with [the equipment] maintaining network connection [which] slowed [service] workflow.” (Facility 1_IF03c)

One facility had not re-implemented after there was a policy change disallowing the practice to continue (External Policy & Incentives > Not Re-Implemented). There was no qualitative data explaining why the 6^th facility did not re-implement.

Not Sustaining Facilities: Facilities that were previously sustaining

In contrast, two of the currently not sustaining facilities (N = 8, 25% of not sustaining facilities) had a previous status of sustained or partially sustained. Representatives from these facilities reported a lack of sustainment occurred in the previous year due to losing necessary employees (Engaging Key Stakeholders > Not Sustained) or finding that the practice was ineffective in their community (Community Characteristics > Not Sustained).

“One of our [employee] positions has been vacant since January and the other [employee] position was realigned under a specific specialty care service.” (Facility 3_IF01a)

Not Sustaining Facilities: Plans to Re-Implement Practice

To better understand the fluid nature of sustainment, facilities that were not sustaining their practice were given a follow-up survey question to determine if they intended to re-implement their practice in the future. Three of the eight (38%) not sustaining facilities intended to re-implement their practice in the future (two previously not sustaining facilities and one newly not sustaining facility) (see Table 5).

Two of these facilities explained that while they had lost necessary staffing, they were in process or planning to replace them to re-implement in the future (Engaging: Key Stakeholders > Not Sustained).

“We recently hired a new Provider and are in the processes of getting her setup with [service] access/equipment.” (Facility 2_IF02b)

Table 5

Plans to Reimplement in Not Sustaining Facilities: Number (Percent) of Facilities

Response	Yes	No	Total Not Sustained
Number (%)	3 (37.5)	5 (62.5)	8

Secondary Outcomes

The following sections describe results from secondary outcomes, which were used to contextualize the primary outcome. Of note, there was a high level of missing data for the secondary outcome questions; our branching logic omitted secondary outcome questions for facilities that did not have their practice in place, i.e., did not re-implement or sustain, including two facilities that were reclassified from Not Sustained to Sustained: COVID-Hold (see Table 6 and 7; Footnote §). As a result, only practice effectiveness and practice institutionalization are presented below. The branching logic is illustrated in Figure 1; reclassification of outcomes is illustrated in Figure 2.

Practice Institutionalization

Overall, there was a high level of concordance (96%) between sustainment and institutionalization outcomes (see Table 6). In addition, two of the three facility representatives that reported partial institutionalization also reported partial sustainment, reflecting initial concordance; however, those two facilities were reclassified from Partially Sustained to Sustained: COVID-Hold during analysis (see Table 6, Foot Note † and Figure 2).

Though less frequent, three facilities had discordant sustainment and institutionalization outcomes. The qualitative data from the survey provided additional context to explain some of the reasons for this discordance. For example, the facility representative that reported partial sustainment (see above) reported the practice was institutionalized where the practice was in use, but it was only in use “in some specialty clinics, palliative care and hospice” (see Table 6, Footnote *). Another facility representative reported the practice was sustained but not institutionalized; though the practice was in use where it was initially implemented, they stated “we want it to expand" (Facility 4_IF02a) (see Table 6, Footnote ‡).

Table 6

Concordance of Practice Sustainment and Practice Institutionalization: Number (Percent) of Facilities

	Sustained: Ongoing and COVIID-Hold	Partially Sustained	Not Sustained	Never Implemented	Total Non-Missing
Institutionalized	23 (95.8)	1 (4.2)*	0 (0.0)	0 (0.0)	24
Partially Institutionalized	3 (100.0)^†	0 (0.0)	0 (0.0)	0 (0.0)	3
Not Institutionalized	1 (100.0)^‡	0 (0.0)	0 (0.0)	0 (0.0)	1
Missing	2 (15.4)^§	0 (0.0)	8 (61.5)	3 (23.1)	13
Total	29 (70.7)	1 (2.4)	8 (19.5)	3 (7.3)	41
* Practice was not in use across all services, but was institutionalized where it was in place (Facility 3_IF04) † Lack of concordance for 2/3 facilities due to reclassification of sustainment outcome (Facilities 2_IF07a, 4_IF05); the third facility did not have any qualitative data to contextualize the responses Facility 4_IF09c) ‡ Practice was sustained but had not spread (Facility 4_IF02a). § Branching logic omitted the institutionalization question for facilities that never implemented/did not sustain, including for two facilities that were reclassified from Not Sustained to Sustained: COVID-Hold.

Practice Effectiveness

Of the 29 facilities sustaining their practice, 23 representatives (79.3%) reported the practice was demonstrating effectiveness (see Table 7). They reported using a variety of measures appropriate to their practices to track effectiveness, including patient-level (e.g., clinical measures, satisfaction rates), employee-level (e.g., turnover rates), and system-level metrics (e.g., time and cost savings). For example, one facility representative reported their practice led to a “decrease[d] LOS [length of stay for patients in the hospital] and higher patient satisfaction scores.” (Facility 4_IF07b).

One representative (N = 29, 3.4%) reported the practice was partially demonstrating effectiveness, stating they had received feedback from employees that the practice was not fully meeting their needs and they were considering adapting the practice to make it more effective at their facility (Facility 2_IF07a) (see Table 7, Footnote *). Two representatives (N = 29, 6.9%) reported the practice was not demonstrating effectiveness; one representative reported the practice “was found to be ineffective with our non-traditional patient population” and they were “transitioning to new presentation and process,” (Facility 4_IF09c) while the other reported they were “not tracking” and therefore were not able to demonstrate effectiveness (Facility 4_IF02a) (see Table 7, Footnote ‡).

Table 7

Practice Effectiveness: Number (Percent) of Facilities

	Sustained: Ongoing and COVID-Hold	Partially Sustained	Not Sustained	Never Implemented	Total Non-Missing
Yes	23 (79.3)	0 (0.0)	0 (0.0)	0 (0.0)	23
Partially	1 (3.4)*	1 (100.0)^†	0 (0.0)	0 (0.0)	2
No	2 (6.9)^‡	0 (0.0)	0 (0.0)	0 (0.0)	2
Missing	3 (10.3)^§	0 (0.0)	8 (100.0)	3 (100.0)	14
Total	29	1	8	3	41
* This facility received feedback from employees and were considering adapting the practice to make it more effective at their facility (Facility 2_IF07a) † This facility did not provide any qualitative data on this question (Facility 3_IF04) ‡ One facility found the practice to be ineffective (Facility 4_IF09c) and the other was not tracking (Facility 4_IF02a) § Branching logic omitted the effectiveness question for facilities that never implemented/did not sustain, including for two facilities that were reclassified from Not Sustained to Sustained: COVID-Hold. The final facility in this category was also a COVID-Hold facility.

With the growing attention on sustainment of EBPs, there is a need for clarity in defining and measuring sustainability versus sustainment. Given that funding often ends before longer-term sustainment can be assessed, it is important for researchers to develop pragmatic sustainment measures that can be used when there are fewer resources and incentives for participants. As part of an ongoing evaluation of the VHA DoE, we developed and administered a pragmatic survey to assess ongoing sustainment across diverse practices. Based on the relatively high response rate (over 60%) and logical responses provided, we can discern several pragmatic features: it was short, easy to understand, and applicable across a wide range of practices [13,20].

Survey results indicated a high rate (over 70%) of practice sustainment among responding facilities, which suggests that the VHA DoE is a promising large-scale model of diffusion. Sustainment rates increased across Cohorts 1 – 4, with later cohorts reporting higher rates of sustainment than earlier cohorts. Ongoing enhancements made to the VHA DoE processes over time (e.g., refining methods to select Promising Practices, better preparing facilities for implementation) may have helped improve sustainment rates over time. It’s also possible lower rates in Cohorts 1 – 2 (2016 and 2017) highlight challenges to sustainment over longer periods. However, only two additional facilities discontinued their practice in the year prior to the survey and these were part of Cohort 3 (2018). Future sustainment surveys with these and new cohorts will help build understanding about changes over time and factors that help or hinder ongoing sustainment. Our ability to continue following these practices is a unique strength of this evaluation.

There were several important lessons learned that will improve our ongoing evaluation efforts and subsequent surveys. First, our primary measure failed to capture nuance in the data related to practices being temporarily on hold. Our survey was administered during the COVID-19 pandemic, during which the CDC issued guidance to cancel and/or reschedule non-essential in-person healthcare. As a result, several respondents used the open text boxes to explain that their practice was in place but on hold during the pandemic, and that they planned to resume operations in the future. However, facility representatives were not consistent in how they answered the primary question; responses ranged from sustained to partially sustained to not sustained. Based on open text explanations, we reclassified some responses as Sustained: COVID-Hold (see Figure 2).

Though temporary holds were common in our evaluation due to the pandemic, EBPs may be paused for a variety of reasons that do not necessarily indicate discontinuation and lack of sustainment. For example, two facility representatives reported their practice was not sustained because they lost employees, but they were in the process of re-hiring; in effect, though the reason was different, these practices were on hold similar to practices paused by the pandemic. It is important to note that turnover and gaps in staffing aligns with a key finding from our earlier work: when implementation and sustainment are achieved via the efforts of a single key employee, it is impossible to reliably sustain the practice when that person leaves or simply takes vacation [18].

In the future, we will add responses to capture whether the practice has been discontinued permanently or is temporarily not in use/not in place. In addition to better fitting the data, this refinement allows the measure to be used at any time point from initial adoption to sustainment; although adoption, implementation, and sustainment are defined differently based on the measurement point, they all assess whether the innovation is being used or delivered. This refinement further shortens the survey by eliminating the need for a follow-up question about re-implementation of the practice.

Second, the sustainment literature often conflates sustainment determinants with sustainment outcomes. Table 1 lists measures conceptualized as outcomes in the literature that were included in our survey. However, if a facility representative reported the practice was not in use (our primary outcome), many of the secondary outcomes were not applicable to that facility. For example, if a practice was not in use, asking whether the practice was demonstrating effectiveness would be illogical; continued effectiveness is a determinant to successful sustainment, not an outcome. Since we did not include secondary outcomes for those who reported they were not sustaining their practice, there was a high rate of missing data for these items by design. Future versions of the survey will reconceptualize Items 3 – 7 in Table 1 as sustainment determinants. Item 2 (Practice Institutionalization) was correlated with our primary sustainment outcome (Item 1). Goodman and Steckler define institutionalization as the “long-term viability and integration of a new program within an organization” [24]. Institutionalization is conceptualized as a deeper, more mature form of sustainment; where the practice is fully routinized and embedded into clinical practice, beyond just relying on the effort of a single person [25]. Basic sustainment (whether a practice is in use) would be a prerequisite for practice institutionalization. Finally, Item 9 (Practice Spread/Diffusion) will be conceptualized as a diffusion outcome. Rogers defines diffusion as “the process through which an innovation […] spreads via certain communication channels over time” [26] within and across organizations [27]. Survey respondents may report sustainment within their own setting with or without diffusion to additional sites. A key goal for the DoE is to widely diffuse effective practices across clinical settings within and outside VHA.

Third, we used “please explain” as a prompt for our open text boxes to provide respondents with an opportunity to contextualize their experiences. However, the information they provided often focused on the rationale for the response rather than barriers and facilitators that led to their reported outcome. For example, when a facility representative reported a practice was sustained, they provided a rationale for their answer (e.g., all core components were in place) vs. a description of facilitators that allowed them to sustain their practice (e.g., continued funding). Changing this prompt to “Why?” and reconceptualizing Items 3 – 7 of our survey as sustainment determinants (see above) will more directly assess relevant barriers and facilitators.

Fourth, we will add a sustainability question (i.e., elicit prospects for continued sustainment) to the survey for all respondents. Although we asked not sustaining facilities a prospective question about plans to re-implement, we did not ask sustaining facilities a prospective question about continued sustainment. Our previous work indicated that predictions of sustainment were relatively accurate [18]. Sustainment is dynamic and may ebb and flow over time; those working most closely with the practice are best positioned to assess prospects for future sustainment as well as anticipated barriers. Low ratings of sustainability could provide an opportunity for early interventions to stave off future failure to sustain.

Limitations

There are several limitations to this evaluation. First, missing data may inflate or skew our sustainment results. The rate of missing data generally decreased with each new cohort, which may be a function of shorter time periods elapsed since initial implementation. Nonetheless, we plan to continue including non-responding facilities in future surveys until they have been lost to follow-up for three years. Second, the branching logic for facilities that did not sustain their practice resulted in a low number of responses to our secondary outcomes by design. We plan to modify future surveys to include determinants of sustainment (Items 3 – 7 in Table 1) regardless of sustainment status.

The development of our pragmatic survey allowed us to assess the ongoing sustainment of DoE practices, demonstrating that the DoE is a promising large-scale model of diffusion. We provide further clarity for concepts of sustainability and sustainment and how each are measured. Given our survey was used with a diverse portfolio of practices, it may serve as a useful instrument for other evaluation efforts.

Consolidated Framework for Implementation Research: CFIR

Diffusion of Excellence: DoE

Evidence-based practice: EBP

Veterans Health Administration: VHA

Ethics approval and consent to participate

Per regulations outlined in VHA Program Guide 1200.21, this evaluation has been designated a non-research quality improvement activity.

Consent for publication

Not applicable.

Availability of data and materials

The datasets generated and/or analyzed during the current evaluation are not available due to participant privacy but may be available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This evaluation was funded by the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI) [PEC-17-002] with additional funding subsequently provided by the VHA Office of Rural Health through the Diffusion of Excellence (DoE).

Authors' contributions

All authors were engaged in the national evaluation of the Diffusion of Excellence (DoE). GJ, LD, and SC designed and supervised the overall evaluation. LD, CR, and AN led data collection, analysis, and manuscript writing for this aspect of the evaluation. All authors were involved in the critical revision of the manuscript for intellectual content. All authors read and approved the final manuscript.

Acknowledgements

The opinions expressed in this article are those of the authors and do not represent the views of the Veterans Health Administration (VHA) or the United States Government. The authors would like to thank Ms. Elizabeth Orvek, MS, MBA for help programming the survey and Ms. Jennifer Lindquist, MS and Mr. Rich Evans, MS for statistical analysis support. In addition, the authors want to express their sincere gratitude to the VHA employees who participated in this evaluation and shared their experiences with us.

Birken SA. Advancing understanding and identifying strategies for sustaining evidence-based practices: a review of reviews 2020:13.
Lexico Dictionary. Determinant. n.d.
Fortune-Greeley A, Nieuwsma JA, Gierisch JM, Datta SK, Stolldorf DP, Cantrell WC, Ethridge AK, MD Angel C, Millspaugh D, Bauch SL, Jackson GL. Evaluating the Implementation and Sustainability of a Program for Enhancing Veterans’ Intimate Relationships. Mil Med 2015;180:676–83.
Stolldorf DP, Fortune-Britt, AK, Nieuwsma JA, Gierisch JM, Datta SK, Angel C, Millspaugh D, Jackson GL. Measuring sustainability of a grass-roots program in a large integrated healthcare delivery system: The Warrior to Soul Mate Program. J Mil Veteran Fam Health 2018;4:81–90.
Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health 2019;7:64. https://doi.org/10.3389/fpubh.2019.00064.
Lexico Dictionary. Outcome. n.d.
Jackson, G.L.; Cutrona, S.L.; White, B.S.; Reardon, C.M.; Orvek, E.; Nevedal, A.L.; Lindquist, J.; Gifford, A.L.; White, L.; King, H.A.; DeLaughter, K.; Houston, T.K.; Henderson, B.; Vega, R.; Kilbourne, A.M.; Damschroder, L.J. Merging Implementation Practice and Science to Scale Up Promising Practices: The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) Program. Jt Comm J Qual Patient Saf 2021;47:217–27. https://doi.org/10.1016/j.jcjq.2020.11.014.
Chambers, D. Building a Lasting Impact: Implementation Science and Sustainability 2013.
Palinkas LA, Spear SE, Mendon SJ, Villamar J, Reynolds C, Green CD, et al. Conceptualizing and measuring sustainability of prevention programs, policies, and practices. Transl Behav Med 2020;10:136–45. https://doi.org/10.1093/tbm/ibz170.
Lennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci 2018;13:27. https://doi.org/10.1186/s13012-017-0707-4.
Scheirer MA, Dearing JW. An Agenda for Research on the Sustainability of Public Health Programs. Am J Public Health 2011;101:2059–67. https://doi.org/10.2105/AJPH.2011.300193.
Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health 2020;8:134. https://doi.org/10.3389/fpubh.2020.00134.
Moullin JC. Advancing the pragmatic measurement of sustainment: a narrative review of measures 2020:18.
Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality Enhancement Research Initiative Implementation Roadmap: Toward Sustainability of Evidence-based Practices in a Learning Health System. Med Care 2019;57:S286–93. https://doi.org/10.1097/MLR.0000000000001144.
Rycroft-Malone J, Harvey G, Kitson A, McCormack B, Seers K, Titchen A. Getting evidence into practice: ingredients for change. Nurs Stand 2002;16:38–43.
Clancy C. Creating World-Class Care and Service for Our Nation’s Finest: How Veterans Health Administration Diffusion of Excellence Initiative Is Innovating and Transforming Veterans Affairs Health Care. Perm J 2019;23. https://doi.org/10.7812/TPP/18.301.
Vega R, Jackson GL, Henderson B, Clancy C, McPhail J, Cutrona SL, Damschroder LJ, Bhatnagar S. Diffusion of Excellence: Accelerating the Spread of Clinical Innovation and Best Practices across the Nation’s Largest Health System. Perm J 2019;23. https://doi.org/10.7812/TPP/18.309.
Nevedal AL, Reardon CM, Jackson GL, Cutrona SL, White B, Gifford AL, et al. Implementation and sustainment of diverse practices in a large integrated health system: a mixed methods study. Implement Sci Commun 2020;1:61. https://doi.org/10.1186/s43058-020-00053-1.
Hunter SB, Han B, Slaughter ME, Godley SH, Garner BR. Predicting evidence-based treatment sustainment: results from a longitudinal study of the Adolescent-Community Reinforcement Approach. Implement Sci 2017;12:75. https://doi.org/10.1186/s13012-017-0606-8.
Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med 2021;11:11–20. https://doi.org/10.1093/tbm/ibz164.
Azam SA, Myers L, Fields BKK, Demirjian NL, Patel D, Roberge E, et al. Coronavirus disease 2019 (COVID-19) pandemic: Review of guidelines for resuming non-urgent imaging and procedures in radiology during Phase II. Clin Imaging 2020;67:30–6. https://doi.org/10.1016/j.clinimag.2020.05.032.
Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qual Health Res 2005;15:1277–88. https://doi.org/10.1177/1049732305276687.
Saldana, J. The coding manual for qualitative researchers. 2nd ed. SAGE; 2015.
Goodman, R.M.; Steckler, A. A framework for assessing program institutionalization. Knowl Soc Int Journral Knowl Transfe 1989;2:57–71.
Zakumumpa, H., Kwiringira, J., Rujumba, J., & Ssengooba, F. Assessing the level of institutionalization of donor-funded anti-retroviral therapy (ART) programs in health facilities in Uganda: implications for program sustainability. Glob Health Action 2018;11. https://doi.org/10.1080/16549716.2018.1523302.
Rogers, E.M. A Prospective and Retrospective Look at the Diffusion Model. J Health Commun 2004;9:13–9. https://doi.org/10.1080/10810730490271449.
Lundblad, J.P. A review and critique of rogers’ diffusion of innovation theory as it applies to organizations. Organ Dev J 2003;21:50–64.

AdditionalFile1Cohort14PracticeDescriptions.pdf
Additional File 1: Cohort 1 – 4 Practice Descriptions. This file provides descriptions of each of the Promising Practices that were included in this evaluation. (PDF 145 KB)
AdditionalFile2SustainmentSurvey.pdf
Additional File 2: Sustainment Survey. This file is the survey that was administered to participants in this evaluation. (PDF 44 KB)

Sustainment of Diverse Evidence-Based Practices Disseminated in the Veterans Health Administration (VHA): Development and Administration of a Pragmatic Instrument

Archived Versions:

Version 1

Abstract

Figures

Contributions To The Literature

Background

Evaluating Sustainment of Evidence-Based Practices is Challenging

Limitations to Current Sustainment Instruments

Veterans Health Administration: Diffusion of Excellence

Methods

Survey Development

Terms and Definitions

Primary Outcome

Secondary Outcomes

Additional Survey Questions

Data Collection

Data Analysis

Primary Outcome

Secondary Outcomes

Results

Primary Outcome

Sustaining Facilities

Sustaining Facilities: Ongoing

Sustaining Facilities: COVID-Hold

Partially Sustaining Facility

Not Sustaining Facilities

Not Sustaining Facilities: Facilities that were previously not sustaining

Not Sustaining Facilities: Facilities that were previously sustaining

Not Sustaining Facilities: Plans to Re-Implement Practice

Secondary Outcomes

Practice Institutionalization

Practice Effectiveness

Discussion

Conclusions

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Supplementary Files

Archived Versions:

Version 1