Forty-two unique clinicians participated in interviews at six months (n=32), 12 months (n=27), and sustainability (n=19). Participants represented multiple different disciplines; across sites the most common were neurology, emergency medicine, pharmacy, and nursing.
General and site-directed activities by phase
PREVENT employed multi-tiered EF by two groups linked by planned feedback loops: QI facilitators and the national implementation support team (see Figure 1). QI facilitators included a QI nurse and the project PI, an internal medicine physician (QI physician), both of whom provided site-directed support. The QI nurse was the primary point of contact for sites, and facilitated site use of program components, assisted with problem solving, conducted outreach, provided logistic support, and offered encouragement and praise in coordination with and under the direction of the QI physician. Members of the interdisciplinary national implementation support team had expertise in data or implementation science (referred to as Data Core and Implementation Core), were responsible for ongoing evaluation of PREVENT, and provided comments and observations mostly related to the QI facilitator site-facing support. Groups met each week and after other activities with sites (e.g., monthly collaborative calls, periodic site visits) to discuss site progress and make decisions about and plan EF.
The QI nurse and QI physician served separate, though coordinated and sometimes overlapping EF roles (see Additional file 1). Both the QI nurse and QI physician built trust with local QI team members, and encouraged and recognize participant success. Together they extensively planned facilitation activities which were usually delivered by the QI nurse.
Site-Directed External Facilitation Activities and Site Implementation
On average, sites received 24 episodes of site-directed facilitation prior to and during active implementation. The QI nurse was involved in many more site-directed facilitation episodes than the QI physician (see Table 1). Both engaged in preparation and planning with sites (e.g., for kickoffs or collaborative calls), promoted networking (e.g., encouraged people to get together), monitored processes (e.g., elicited information about site implementation for the Implementation Core), and provided education (e.g., sent scholarly citations) (see Additional file 3 for excerpts from communications). In her site-directed facilitation, the QI physician was more likely than the QI nurse to engage stakeholders (e.g., gain buy-in from site leadership), perform audit and feedback (e.g., give data from the Data Core), and brainstorm solutions (e.g., offer suggestions for local care gaps), and to do these things during pre-implementation. Across sites, 30% of the site-directed EF was initiated by site champions.
Site-directed EF varied in quantity and type, though overall networking (in 45% of the site-directed EF episodes) and preparation and planning (44%) were the supports most often provided (Table 2; see also Additional file 4 for more detail). Site 103 was at the highest end of most metric ranges: they had the most site-directed EF episodes (39), their champion initiated 54% of their episodes, and they had the highest percentage of episodes that included process monitoring (49%), audit and feedback (33%), and brainstorming solutions (31%). This pattern may reflect the rapport that developed between the QI nurse and the champion at Site 103, as well as the champion’s stated strong motivation to improve Veteran care. Site 104 had the fewest number of site-directed EF episodes (21), no contacts initiated by the site champion, and had the lowest percentage of episodes that included preparation and planning (38%), networking (38%), brainstorming solutions (10%), and audit and feedback (0%). Early in implementation, local team members requested information to aid product development, but most of the site-directed interaction related to monthly collaborative calls and addressing a technology issue.
There was also variability in EF activity within sites over time (Figure 2; for more information, see Additional file 4). Sites varied in terms of when they received more or less site-directed EF. In some cases, more EF corresponded to increased site implementation activity (e.g., to support use of data to direct activities), in other cases it occurred during site inactivity (e.g., to suggest solutions to challenges). Across sites, there was a temporal trend for fewer site-directed education EF across the active implementation period. Facilitators often spent more time during early quarters clarifying and explaining the PREVENT program and helping sites develop local education materials.
Perceptions of EF by QI team members over active implementation from the qualitative interviews
During interviews, participants characterized the QI nurse EF facilitator as accessible, responsive, and helpful. QI team participants described varying types of relationships with the QI nurse, ranging from recognizing her name only from emails, to having interactions with her oriented around specific tasks, to repeated interactions and familiar working relationships. The latter type was almost exclusively described by site champions who were also the points of contact between their local team and the QI facilitators.
Participants reported that the QI nurse provided problem-solving, feedback, and accountability support. She helped participants understand how to effectively use PREVENT components (“Education”), particularly related to data and the Hub: “I liked [the Hub office hours]…that helped a greater understanding for the web site itself and what we can use with it.” (104_12m_3)
She also helped sites problem-solve and, as needed, connected them to experts (“Networking”) to help them overcome challenges (e.g., implementing a clinical informatics tool):
"When we were first putting the bones to the dashboard together and trying to align the Hub data with what we were envisioning for the dashboard, I know that she spent a lot of time contacting folks to get the background information about how the Hub data was being pulled and what that data meant." (105_12m_5)
Many of the QI nurse’s interactions with sites (multiple per month) served to elicited site updates (“Process monitoring”) and included notifications about updated Hub data (“Audit and feedback”). Some participants described how this kept them on track: “it’s not as fun to kind of necessarily maintain something. So you all kind of periodically nudging us is a good thing. … With the nudges come sort of the carrots as well of feedback in terms of how we’re actually doing. Putting them in context is very helpful” (101_12m_4).
“Nudges” helped them prioritize the project, situated the project in terms of practical performance metrics, and often prompted sites to action. For example, one champion said reminders “keeps you on your toes” (104_18m_1). Requests for site updates (sometimes multiple) before collaborative calls encouraged reflection and action. Updates made the QI nurse a source of accountability, while also potentially a trusted source for problem solving:
"I send [QI nurse] any progress that we make on the order sets because I'm constantly worried that she feels like that we’re just going to stagnate... any time that I’ve had like a question about anything or if I wanted to like bounce an idea or ask for feedback, I have like no qualms about emailing [QI nurse] or hitting her up on [instant messaging]." (103_6m_2)
Some site team members also reflected on missed opportunities for support. A member of a local team described deciding not to seek help after encountering a challenge:
“I thought about [talking to QI nurse], but [the site champion] and I were just like well, I think we’ll just move on ... But I know [QI nurse] was available, and I wish I had used her. Maybe I could’ve talked to [QI nurse] earlier when I was getting pushback from Nursing to try to figure out what to do, but I didn’t. And I didn’t know if it would have made any difference. I think it was just internal issues here." (105_12m_4)
Given the more behind-the-scenes-role of the QI physician, it was not surprising that participants described fewer interactions with the her but nonetheless often reported a respect for the QI physician’s expertise. Contacts often concerned with clinical issues (e.g., developing drug interaction guidelines on the statin protocol, preparing lectures); champions reached out to ask for professional slides the QI physician prepared for site use. This site champion illustrated how she utilized multiple tiers of the QI facilitation:
“project planning, it’s not something that I'm used to doing…[QI nurse] was very helpful. … When I did my grand rounds, I contacted [QI physician], and then on one of the talks about the carotid endarterectomies, I had some questions about I think some articles and stuff and she was very responsive. ... [I had communications with QI physician about] things more medically related things, or if I had questions about something that was discussed.” (101_12m_5)
Critical EF Junctures
Through constant monitoring of feedback loops, the PREVENT facilitation team surveilled for potential implementation problems and discussed during team meetings potential interventions. For two sites, concerns were raised repeatedly and led to interventions by the facilitators at “critical junctures.”(17,25) These are times of tension and uncertainty about the project status (e.g., questions about goals and direction, if and how the project can move past a barrier), where decisions and actions by individuals can lead to changes in implementation trajectory.
Site 106 struggled from the start to implement QI activities (Figure 3). With a rapidly growing patient population, new and expanding programs, and staff turnover, local pressures made it difficult to convene a committed, local team. During the first six months, the QI facilitators and the national implementation support team reflected on and expressed concern about (during internal team briefings) lack of site engagement in the collaborative calls, team development, and implementation progress, and the loss of one of the co-champions. To assist in team building, the QI nurse reached out to the remaining champion to suggest potential local roles who could help. To prompt planning and action, using data provided by the Data Core, the QI physician sent an email that outlined the local site’s data, where that facility was doing well, and processes that could be improved. Prior to the six-month evaluation team visit, a member of the Implementation Core strongly recommended that the visit be used as an intervention to re-engage the site. The QI physician traveled to the facility in person to engage local physicians throughout the site visit. While on site, a member of the Implementation Core enlisted an additional brand-new local champion. In the following months, the new champion was able to quickly engage the participation of another service, began participating in the broader PREVENT community of practice, and took responsibility for working and monitoring progress toward goals. However, the new champion continued to be challenged to engage the needed stakeholders and to understand how to implement PREVENT fully within the constraints of local context.
Site 103’s champion was engaged and highly motivated from the start of implementation (Figure 4). Team debriefings frequently highlighted the implementation support team and QI facilitators’ confidence in the champion’s ability to problem solve challenges and praised her progress, even when noting the lack of local neurology engagement. Early into implementation, the QI nurse offered to invite some of the non-engaged stakeholders to the collaborative call, but the champion did not respond. Nine months into active implementation, the champion emailed the QI nurse discouraged by continued poor process of care metrics. The QI nurse immediately called the site champion to provide encouragement and a positive perspective on the process of care data. Over the next few days, through a series of instant messages and emails, the QI nurse provided patient-level data and helped to re-interpret the data to emphasize improvements in several processes of care and near misses with neurology consultations. The champion quickly responded by re-educating medicine attendings and talking with the neurology chief. The QI physician provided updated data from the Data Core that the champion used in a presentation to the neurology service, and these data helped convince the neurology service to agree to a critical process change. Although the champion continued to struggle with physician engagement throughout the sustainability period, during the 12-month and sustainability interviews she described this EF intervention involving data reframing and offering moral support as critical to enabling her to act:
“I just felt like that I was working so hard, and I was like trying to cover all of these holes… and our compliance rate just ended up being like zero for the month … I really just wanted to quit because I just was really frustrated. … [QI nurse] was really helpful in allowing me to just kind of call and vent, and then she was also really very encouraging at saying like I know that your without fail rate is zero, but you also have to look at the improvements that you did make this month. … that was an interesting lesson to learn that you might feel like that you’re unsuccessful because of that one particular metric, but you have to look at the bigger picture… I took the weekend and came back and felt like okay, let’s tighten up and figure out what we need to address and move forward, and then the next month was better.” (103_12_2)
At the end of active implementation, without the structure of the PREVENT program (e.g. monthly calls) and the reminders from the QI nurse which prompted teams to communicate and act, many local teams stopped having regular check ins. A champion described the impact of losing external facilitation: “basically the net effect was that the group as a system for checks wasn't as…. You know. Wasn't as involved anymore. Right. So then that just or it just kind of went to the individual members to see what their responsibilities were.” (104_Sus_1) The loss of check-in emails led to sharp declines in team activity and a parceling of ownership and activities by role.
Another champion said that the loss of external facilitation during sustainment resulted in less motivation to and information that would help them innovate:
“it’s not like we’re pushing for that innovation, like we’re not getting those ideas to think about to push to get to the next level. So I see that as a deterrent. I mean of course I could research it and learn more about it, and spend a lot of time, but I don’t have a lot of time to do that. So it was good to kind of just get that information pushed to us” (106_Sus_1)
However, not all sites felt the loss in the same way. Participants at site 101 said that because PREVENT had been well integrated into their existing stroke program, the loss of the external facilitation had resulted in PREVENT not being as top of mind and less data sharing among the local group, but had ultimately not been that impactful: “we’ve continued to, I guess, you say maintain even without the additional support or the constant reminders. Once we got the program situated, seems to be doing pretty well.” (101_Sus_1)