Eighty-two implementation leads, who were members of DoE Cohorts 1–5, collectively implemented 57 different Promising Practices (25 clinical interventions, 23 process improvements, 9 staff-oriented interventions). Of the 57 practices, 53% had a virtual component. See Practice Descriptions in Additional File 1.
As of 2021 (timepoints ranged from 1–5 years after a 6-9-month initial implementation period), about one-third of leads reported their Practice was fully sustained, one-third reported their Practice was not fully sustained, and one-third did not respond. One lead was missing outcomes for all time-points. Twenty-nine percent (24/82) of leads reported consistent outcomes from initial implementation to sustainment, 43% (35/82) of leads reported inconsistent outcomes from implementation to sustainment, and pathways for 28% (23/82) of leads were unknown because they did not respond to the 2021 survey. Tables 3–4 list facilitators and barriers to sustainment for the 66% of leads who responded to optional open-ended text boxes. Additional File 3 provides more details, showing sustainment outcomes separately by Practice type. Figure 1 visually displays all longitudinal pathways, showing shifts in Practice implementation/sustainment status over time.
[Figure 1]
Longitudinal Pathways for Practices that were Fully Sustained in 2021
Thirty-five percent (29/82) of leads reported their Practice was fully sustained in 2021, which was an average of 2.3 years (range: 1–5 years) after implementation. Of these 29 leads, 76% reported full implementation at their 6-9-month milestone; the remaining leads completed their implementation milestone later. Further, of these 29 who reported their practice was fully sustained, 79% (23/29) reported full sustainment at initial follow-up. The six leads who did not initially sustain their practice went on to sustain their Practice by 2021.
Facilitators of sustainment included demonstration of Practice effectiveness/benefit, sufficient organizational leadership, appropriate workforce, and Practice adaptation/alignment. Leads also described potential barriers to future sustainment, including workforce turnover, challenges with critical incidents related to the COVID-19 pandemic, and concerns about ongoing support; they also listed potential facilitators for future sustainment that included having appropriate workforce and sufficient organizational leadership and ongoing support. See Tables 3–4 for all factors influencing sustainment. Sustained Practices included more clinical interventions (45%, 13) and process improvements (45%, 13) compared to staff interventions (10%, 3) and were almost evenly split between the presence (48%, 14) or absence (52%, 15) of a virtual component. Figure 2 displays longitudinal pathways for leads that fully sustained their practice.
[Figure 2]
[Table 3]
Consistent Pathways
Among the 29 leads who reported their Practice was sustained in 2021, 66% (19/29) consistently sustained; meaning that they had sustained at all timepoints after fully implementing. All leads anticipated that their Practice would continue to be sustained into the future and 89% (17/19) reported their Practice was institutionalized and effective. See Fig. 3 for a consistently successful pathway showing outcomes with qualitative explanations.
[Figure 3]
Inconsistent Pathways
Among the 34% percent (10/29) of leads with an inconsistent pathway to sustainment in 2021, the majority experienced initial challenges with implementation (n = 6) compared to challenges with implementation and sustainment (n = 1) or were missing data (n = 3) in 2020. Leads with inconsistent pathways to sustainment needed more calendar time than the 6-9-month facilitation period to overcome implementation barriers, which were often related to insufficient workforce and available resources. One lead also reported a temporary pause in sustainment due to critical incidents related to the COVID-19 pandemic that were resolved when pandemic restrictions at their facility were loosened.
Once achieving sustainment, these leads emphasized that facilitators to sustainment were sufficient workforce and organizational leadership. All but two leads anticipated future sustainment. One lead anticipated future sustainment as “unlikely” without explanation and one did not respond to the question (see Fig. 2). Most (8/10) leads described their Practice as institutionalized and effective in 2021. Two leads described partial institutionalization because of no/limited funding or lack of adequate number of service users (i.e., insufficient Veteran enrollment in a voluntary program). Two leads reported their Practice was partially effective: one of whom cited no/limited funding as an issue. Figure 4 provides an example pathway showing outcomes and qualitative explanations.
[Figure 4]
Longitudinal Pathways for Practices that were Not Fully Sustained in 2021
Thirty-seven percent (30/82) of leads reported that their Practice was not fully sustained because they were in a liminal stage (neither sustained nor discontinued) or permanently discontinued as of 2021, which was an average of 2.1 years (range: 1–5 years) after implementation. Only 43% (13/30) of these leads reported full implementation at their 6-9-month milestone with five additional leads completing implementation later. Only 23% (7/30) of leads reported full sustainment at initial follow-up.
Barriers to sustainment included: insufficient workforce (losing or not being able to hire staff), not being able to maintain EIP fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of EIP, no ongoing support, lack of trained personnel to continue the EIP, and/or EIP effectiveness/benefit was not observed. Despite not being fully sustained, these leads also described facilitators to sustainment. The most frequently reported facilitators to sustainment were internal/external EIP champions and sustained/attention to topic/priority, which were not mentioned by leads with sustained practices. See Tables 3–4 for all factors influencing sustainment. Practices that were not fully sustained had a similar percent of clinical interventions (50%, 15), process improvements (43%, 13), and staff-oriented interventions (7%, 2) as those that were sustained but had fewer Practices with virtual components (37%, 11). The following sections describe results for leads who reported un-sustained Practices, which is organized by status (liminal sustainment or discontinued permanently) and pathway (consistent or inconsistent). Figure 5 displays longitudinal pathways for leads that did not fully sustain their practice.
[Figure 5]
Liminal Sustainment
Eighteen percent (15/82) of leads reported that their practices were not fully sustained because they were in a liminal stage of sustainment (6 partially sustained, 9 temporarily paused) since they were neither sustained nor discontinued in 2021. The major barriers associated with practices that were in a liminal stage of sustainment included insufficient workforce, no ongoing support, lack of trained personnel, and critical incidents related to the COVID-19 pandemic. Though fewer facilitators were mentioned compared to barriers, the top facilitator to sustainment was internal/external EIP champions.
Despite their liminal status, almost half (7/15) of these leads were optimistic about sustaining their Practice in the future. However, the remaining leads did not expect to sustain their practice (5) or were uncertain about future sustainment (3). Most of these leads reported their Practice was not fully institutionalized (8 no, 4 partial) nor effective (3 no, 8 partial).
Permanently Discontinued
Eighteen percent (15/82) of leads reported that their Practices were not fully sustained because they were permanently discontinued. Common barriers associated with Practices that were discontinued included two of the same as those in a liminal stage (workforce and critical incidents). However, not able to maintain EIP fidelity/integrity (top barrier), organizational leadership did not support sustainment of the EIP, and system policy change were cited as other important reasons for Practice discontinuation. Despite Practices being discontinued, two leads cited sustained attention to topic/priority as a facilitator.
Among leads who provided responses to secondary outcomes (6/15), half reported their Practice was not fully institutionalized (2 no, 1 partial). Unexpectedly, the remaining leads with discontinued Practices (3) reported their practice was institutionalized due to some aspect of the Practice becoming routinized. Regarding effectiveness, more leads reported their Practice was not fully effective (3 no, 1 partial). However, those who reported their discontinued Practice had demonstrated effectiveness (2), cited they were tracking an aspect of Practice effectiveness.
[Table 4]
Consistent Pathways
Only 17% (5/30) of leads reported consistently less successful implementation and sustainment outcomes over time. Two leads reported their Practice being partially implemented and sustained through 2021. Despite these leads’ consistent liminal status, responses to secondary outcomes of institutionalization, effectiveness, and anticipated sustainment were different from each other. One lead reported partial institutionalization and effectiveness and anticipated full sustainment in the future, but it was dependent on having sufficient workforce in place. The other lead reported that the Practice was effective but was not institutionalized and would not be sustained in the future due to insufficient workforce.
The other 3/5 leads who were consistently less successful did not implement their Practice by the end of the 6-9-month facilitation period and then reported their Practice was permanently discontinued. Only 1/3 leads responded to the institutionalization and effectiveness outcome questions and responded that their Practice was not institutionalized nor effective. These leads experienced insurmountable barriers with implementation and never reached the sustainment phase because of critical incidents related to the COVID-19 pandemic or no/limited funding. Figure 6 provides an example pathway showing outcomes and qualitative explanations.
[Figure 6]
Inconsistent Pathways
Eighty-three percent (25/30) of leads whose practices did not fully sustain in 2021 reported inconsistent outcomes over time, which meant their outcomes did not align over time and/or they were missing at least one outcome prior to 2021. There were two main types of inconsistent pathways leading to unsuccessful sustainment. The first type consisted of leads who successfully implemented their Practice, but experienced challenges with sustainment. Early on, thirteen leads reported full implementation at the end of the 6–9-month facilitation period and another five reported full implementation with additional time when responding to the follow-up survey. Although 72% (18/25) of these leads were successful at implementation, by 2021, nine downgraded to a liminal stage of sustainment (4 partially sustained, 5 temporarily not sustained) and 9 reported being discontinued permanently.
The second type consisted of leads (7/25) who experienced some challenges early on with implementation (3 not implemented, 2 partially implemented) or were missing implementation outcomes (2). These leads’ status fluctuated up and down over time and by 2021, they all downgraded to being temporarily not sustained (4) or permanently discontinued (3). Figure 7 provides an example pathway showing outcomes and qualitative explanations.
[Figure 7]
Missing Data in 2021
Missing data resulted from leads being lost-to-follow-up because they did not respond to implementation or sustainment survey assessments. Forty-one percent of leads had at least one missing time point across all years of data collection with an average of 1.8 missing time points overall. Only one lead was missing responses for all time-points. Among leads with two missing timepoints in a row, only 2/5 responded to subsequent surveys. Leads from Cohorts 1 (53%) and 3 (43%) had more missing data than those from other Cohorts (Cohort 2 = 24%, Cohort 4 = 19%, Cohort 5 = 17%).
In 2021, 28% of leads (23/82) had missing data. Leads with missing 2021 sustainment outcomes had more process improvement Practices (39%, 9) compared to staff interventions (35%, 8) and clinical interventions (26%, 6), which differed from practices that were sustained or not fully sustained. These leads also had fewer practices with virtual components (30%), which was like those that were not fully sustained.
After the 6-9-month facilitation period, only 17% (4/23) of leads had missing implementation outcomes data. However, 3/4 leads responded at follow-up. One of three leads reported their practice was implemented and sustained before being lost-to-follow-up. Whereas the other two other leads reported at follow-up that their Practice was not implemented nor sustained before being lost-to-follow-up in subsequent years.
For all the other leads (19/23), once they failed to respond to a survey, they were consistently non-responsive. Of the 23 leads lost-to-follow-up in 2021, only two (9%) were lost-to follow-up two years earlier in 2019. However, by 2020, an additional 12 leads (52%) were lost to follow-up, and the remainder (9/23; 39%) had their first missing data in 2021.
Sixteen leads with data missing in 2021 (16/23; 70%) were initially reported as fully implemented by the time of the second implementation assessment. Although these leads were lost-to-follow-up in 2021, most (69%,11/16) reported full implementation or sustainment as their last known status. The remaining leads reported a downgraded status of being temporarily (2/17), partially (1/17), or not (2/17) sustained before being lost-to-follow-up in subsequent assessments.