The six PREVENT sites were geographically diverse including sites in the West, Northeast, Southeast, and Midwest. The annual TIA patient volume varied across PREVENT and controls sites and across periods, ranging from 10 to 67 (Table 1).
The patient characteristics were similar between patients cared for in the PREVENT sites and the matched control sites (Supplemental Table A). Within the PREVENT and control sites, the characteristics of patients were also similar for those cared for during the baseline, active implementation, and sustainment periods (Supplemental Table A).
Overall, the observed without-fail rate at PREVENT sites improved from 36.7% at baseline to 54.0% during active implementation and settled to 48.3% during sustainment. Whereas at control sites, the observed without-fail rate improved from 38.6% at baseline to 41.8% during active implementation and continued to 43.0% during sustainment (Figure 1). Quality of care varied considerably across sites and across time periods (Tables 1 and 2). The without-fail rate improved at 3 sites, declined at two sites, and remained unchanged at one site during sustainment. The without-fail rate during sustainment was not related to baseline or active implementation rates. In adjusted analyses, although the without-fail rate improved at PREVENT sites compared with control sites during active implementation, no statistically significant difference in quality was identified across sites during sustainment (Table 2; Supplemental Table B).
The mixed methods assessment thematic analyses identified two core activities that existed at sites where quality of care improved during sustainment): (1) the importance of integrating processes of care into routine operations and (2) the value of reflecting and evaluating on performance data to plan quality improvement activities or respond to changes in quality. A key challenge during sustainment was competing demands from new facility quality priorities.
Cross site case comparisons indicated that strong clinical champions played an important role by embedding the targeted processes of care into their organization during active implementation to ensure sustainment. Reflecting and evaluating quality performance was common among facilities which improved during sustainment. In addition, sites that improved during sustainment scored highly on the CFIR construct of cosmopolitanism (e.g., voluntarily and intentionally reaching out to work with others outside of their own environments) whereas cosmopolitanism was not observed either among sites without change or with declining quality during sustainment.
Promoting Sustainment: Integrating processes of care into routine operations
The site teams worked to embed TIA quality improvement into a broad range of existing processes or structures of care at their site. For example, two sites embedded TIA quality of care as an ongoing topic into their existing stroke team structure and activities. Three sites described how they provided regular updates to facility leadership about quality of care within the context of regularly scheduled meetings. Sites with improvements in quality during sustainment embedded PREVENT activities within the fabric of routine work-flows:
“…it’s not so much that people are thinking of it as part of PREVENT, it’s just on that’s just what we do for patients.” [Site B_2]
And from a second site which improved during sustainment:
“…we changed the process, and so it’s not something that I have to monitor every day.” [Site F_5]
Several interviewees described automatization of PREVENT processes such that they no longer required active attention or work.
“I think the biggest impact was…allowing things to be able to be somewhat on autopilot because there’s little that the clinicians have to do other than implement it or rather admit the patient. You know it flows very consistently with the existing process.” [Site F_2]
“It was pretty well hard wired into our practices before we graduated the program. As we went through it was minor tweaks and changes so it’s already a regular deep-seated process for us.” [Site F_3]
Promoting Sustainment: Value of examining performance data regularly to identify and respond to changes in quality
The sites that improved or maintained during sustainment used performance data to check on their status. This habit of status-monitoring with performance data, formed during active implementation and persisted into sustainment, was performed by the clinical champion.
“It was good to see the trending of our data points…when we had our group meetings and you know we would discuss those numbers…” [Site A_1]
At one site, the champion used the data to engage front-line staff:
“…sometimes it’s surprising like oh I didn’t know we were like this this last time I looked down and I looked at the Primary Care 30-day visit, and I was wondering oh wow, I didn’t know that was kind of lower than the national average for that…that kind of has given me something to think about to bring to like you know Neurology or like the hospitalist folks that I deal with on different kind of projects.” (Site D_1)
Quality of care data at the facility level allow champions to identify patterns in care quality that may be “invisible” to them (e.g., care provided by other services):
“By having the dashboard, it helps me capture patients that I myself would not necessarily have known about.” [Site B_2]
In contrast, team members at sites with declining quality of care during sustainment reported that they believed that PREVENT was integrated into existing structures or policies at the facility but did not actively examine performance data during sustainment, and were therefore, unaware of evidence of decrements in performance during sustainment.
“I think that all of the things that we did do are still in place today. So that’s really good.” [Site C_2]
“I guess that I haven’t been as hands on with like the sustainability portion of the study. Since we haven't met as a team since we graduated, I can't really speak on how well that we’ve sustained. I still think that we’re all taking care of TIA patients and things like that, but as far as looking over the seven criteria as a team together, we haven't done that…Probably just not paying attention I guess. [Site C_3],
“…the changes that we have implemented have been sustained. So the same changes have been maintained throughout the facility after the graduation.” [Site E_2]
Barriers to Sustainment: Difficulties using performance data to inform quality improvement
Respondents at sites without an improvement in quality during sustainment identified two challenges related to using performance data for quality improvement: (1) the difficulty of interpreting pass or fail rates in the setting of a low volume of patients, and (2) coding problems that made it difficult to discern if without-fail rates reflected genuine systemic issues with patient care or alternatively were spurious.
“…the biggest barrier that we’ve faced is just coincidence in numbers. This year, our TIA numbers have actually been less than in past years…The most that we’re able to do is review the cases and see…where things went wrong and what we could do to fix it…it’s really hard to identify patterns when the pattern is like we have one patient to review. Like sometimes it’s a little easier when you see like hey, if we had like ten patients, we’d be like all right. Eight were fine, but the pattern of the people who did have issues was this. But I think that that’s what’s really been the toughest part.” [Site C_1]
Concerns about miscoding made data more challenging to interpret:
The weakness of it [local PREVENT] is again, like there have been a few cases where I think that either their TIA was mislabeled and we didn't like push or emphasize it strongly enough… [Site E_1].
During active implementation, some site team members identified relatively infrequent coding issues which nonetheless influenced their performance data. For example, a patient coded as having atrial fibrillation and identified as failing the anticoagulation for atrial fibrillation metric may not actually have had atrial fibrillation, and therefore it was appropriate not to have prescribed an anticoagulant. Given that the without-fail metric was based on administrative data, coding errors required chart review for identification, and working with facility coders for remediation. During the active implementation period, external facilitators conducted chart review for patients cared for at participating sites and helped local site teams interpret the performance data that were provided in the hub in the context of the chart review information; but this assistance was not routinely available during sustainment.
Barriers to Sustainment: Competing Demands
Respondents across sites identified the problem of competing demands on time as the major threat to sustainment.
“I mean I think that it’s a great program. You know. To come together with like a common goal and actually see how good the results can be. I think that it’s an awesome program. I just wish that there was a way for us to I guess keep everything going. I mean I know that we were still keeping things going, but it’s like they’ll come to us and say we’re trying to improve CHF care, and then we’ll do that, and then it’s like going on to the next project.” [Site E_3]
“I haven't been able to do anything with the sustainability portion, like I said, just because there are a ton of other projects, and I don’t want it to sound like that I'm giving an excuse, but there is literally like a million other things that management focuses on or that comes up in the ED.” [Site C_3]
However, some clinical teams in sites that successfully sustained their quality were able to overcome competing demands by modifying their culture. By engraining the processes involved in both providing and monitoring high quality care into routine practices, oversight require less active engagement and therefore the issue of competing demands was less salient.
“…we changed the process, and so it’s not something that I have to monitor every day. … So I think that the culture, and also we changed the culture. [Site F_5]