Our research findings explain important contextual factors and mechanisms that had a perceived effect on the sustainability of two provincially scaled, multi-component interventions. The discussion that follows outlines four key mechanisms that were perceived by our participants to facilitate intervention sustainability. These mechanisms include implementation strategies of collaboration, audit and feedback, informal leaders, and patient stories.
Learning collaboratives as a mechanism for sustainability
Collaborative research approaches are becoming increasingly used by healthcare systems, research funders and government organizations as part of health services research andimplementation science (33). A collaborative research approach provides the opportunity for patients, healthcare providers and other key stakeholders to be active participants in the design process rather than the traditional approach of being a passive recipients of design work (i.e. intervention) (34). Participants from both cases discussed LCs as the key mechanism for a collaborative approach that facilitated intervention sustainability. In accordance with the Dynamic Sustainability Framework (17) our findings suggest that active partnership among all relevant stakeholders is essential to sustaining interventions within care settings. As in the Consolidated Framework for Sustainability (31), our research highlights the importance of relationships, collaboration, and networks for sustainability.
A LC is an organized, multifaceted approach that includes teams from multiple healthcare sites coming together to learn, apply and share improvement methods, ideas and data on performance for a given healthcare topic (35, 36). In our evaluation, LCs occurred in-person for case A with virtual components introduced in case B. While there is clear evidence on the effectiveness of in-person LCs to enhance learning, less is known about the effectiveness of virtual LCs (37). Similar to other research, our findings suggest that creating a culture of continuous learning, promoting accountability, and creating an inter-organizational support network from which sites can learn from others’ successes and challenges are some of the main benefits of LCs (38). Despite the benefits of LCs identified in our study, and others, questions remain about the effectiveness of LCs for behavior change, the use of skills gained in the LCs, the impact of LC for sustained improvement, the effectiveness of LCs as a strategy for sustainability and the and cost-analyses of LCs over time (36, 38, 39).
A systematic review by Wells et al., (36) found that LCs characteristics, such as the number, length, and delivery mode (i.e. virtual vs in-person) varied across studies. This highlights the existing variability in the design and delivery of LCs; there is a paucity of evidence on how best to design and implement a learning collaborative. Similar to Hoekstra et al., (33) we argue the need for research to examine how and why collaborative research approaches and interventions (such as LCs) work, including the key principles, strategies, outcomes, impacts and contextual conditions these approaches function under. This knowledge may allow for more tailored and efficient stakeholder engagement in future.
Continuous monitoring, audit, and feedback for sustained change
Monitoring, audit, and feedback (A&F) of interventions are important strategies to facilitate buy-in, maintain compliance and ensure the continuation of improved outcomes (40). Our findings pertaining to how A&F supports ongoing staff engagement, by hearing, and seeing data in a group atmosphere are well aligned with the literature (40-42).
The use of data to monitor local implementation is not just a means of promoting accountability, but also a mechanism to solve problems that impair performance. In the absence of regular, careful monitoring, implementation may be more liable to fail or revert to previous practices (40). From our findings, it is evident that careful and regular monitoring needs to happen from early implementation of an intervention to support sustainability. Implementation teams and operational leaders need to plan a monitoring, A&F system that makes sense and is meaningful to all of those involved and can demonstrate impact.
Previous research has been done to synthesize the effectiveness of A&F for implementation research. One Cochrane systematic review on 140 studies found that A&F can lead to important improvements in professional practice. However, the effectiveness of A&F as an intervention to change provider behavior depends on both the content of and how the feedback is provided (41). The Dynamic Sustainability Framework (17) suggests that ongoing feedback on interventions should use practical, important measures of progress and relevance. The framework recommends the use of measures that are feasible, relevant to desired outcomes of patients and align with the ‘fit’ between intervention and context. There is a lack of guidance on what dose of feedback and which modalities are most effective to support the sustainability of scaled interventions over time. A&F is most effective when provided more than once (41), however it is unclear from the literature and our study, how often the intervention is required for sustainable impact. Another study that examined the use of theory in A&F studies found that there was an overall lack of use and consistency of explicit theory to guide A&F interventions (42). As a result of these issues, the most important active ingredients and mechanisms that enable successful A&F intervention for healthcare improvement remain unclear (43).
In an effort to bridge this knowledge gap, Ivers et al., (43) provided potential best practice guidance recommendations for A&F interventions in relation to audit components, feedback components, the nature of behavior change required and target, goals and action plan. Taking study findings into account, we concur with these best practice recommendations. Our results further emphasize the presence of variance in contextual factors (e.g., resource allocation), intervention design (e.g., mode of delivery of feedback, frequency of feedback,), recipient characteristics (e.g., profession, role, years of experience) and behavior change characteristics (e.g. readiness for change, practice change) that influence the effect of A&F on sustainability. Future research is needed to examine the process of delivery, effectiveness, and impact of A&F on the sustainability of multi-component, scaled interventions, even in a single provincial system undertaking coordinated, provincial implementation and scale.
The influence of informal leadership for sustainability
Previous implementation research studies have established the influence of formal (e.g., administrators) and informal leaders (e.g., champions) and their activities (e.g., facilitation, support) on sustainability (1, 44, 45). Informal leaders, sometimes referred to as champions, opinion leaders, change agents, or knowledge brokers, are considered front‐line practitioners, driving the implementation of a wide range of change interventions in healthcare settings (46-48).
A focus on informal leaders is essential because this is where the quality of care ultimately affects patient outcomes (49). In alignment with our study, a Cochrane review determined that the effectiveness of informal leaders as an intervention for the implementation of evidence-based interventions appears comparable, or sometimes even superior, to other interventions (50). As in our study, Ennis et al., (51) found that informal leaders contribute to creating a positive work environment. Informal leaders influence workplace culture and have significant impacts on team efficacy and performance by seeking out opportunities to promote, improve and negotiate best care practices (51).
Our findings suggest that front-line informal leaders are valued and play an important role in the implementation and sustainability of multi-component, scaled interventions. In our study, front-line informal leaders were active participants in the intervention and were encouraging and motivating for others. This aligns with existing evidence that informal leaders are effective because they socially influence other professionals, and that this influence is a function of the respect of their peers (48, 50). Furthermore, it was recognized that senior leaders (i.e. executive directors, unit managers) may not necessarily be the best people to promote continuation of interventions due to their lack of understanding of the daily work of front-line staff. Informal leaders were viewed as more influential based on their credibility amongst colleagues. This same phenomenon has been found in similar work (52).
Engaging influential individuals across organizations can help to secure the credibility of interventions and strategies to develop “informal leaders” have shown to be effective in implementing changes at the clinical level (52). Hence, implementation strategies should recognize and seek to engage with and develop individuals who have not traditionally been perceived as leaders. In the later stages of implementation, senior leadership should plan for strategies to help informal leaders emerge, ensuring they have the capacity and capabilities to lead in sustaining efforts. Like the Consolidated Framework for Sustainability Constructs in Healthcare (31) our research highlights the importance of the people involved (e.g., champions) for sustainability.
Impact of sharing patient and family stories
In our initial program theory, we did not hypothesize patient stories as an important mechanism for the sustainability of an intervention. Patient stories have previously shown merit, with reported improvements in care practices, positive staff engagement, a way for staff to “remember why we’re here”, and combat burnout (53, 54). In this study, patient stories provided a way to connect with patients, to understand their experiences, and to remind staff why the intervention was important, facilitating sustainability.
Stories have a degree of emotional power that can spark attention, resonance and change (55-58). Like our findings, other studies have found that sharing patient success stories enables HCPs to feel energized after watching them, as these stories are “impactful, heartwarming, and understandable” (54). Foster et al.,(59) found that listening to patient stories not only had profound emotional effects on HCPs, but motivated practice change as they developed newly formed intentions to improve patient outcomes. Similarly, Haigh and Hardy (60) found that patient stories shown to HCPs led to reflection, empathy and discussions surrounding practice change aimed at service improvement. These studies mirror our findings in that sharing patient stories can influence better service and patient outcomes through staff motivation and reflection of current practice. Despite the clear impact our study, and others, have shown of patient stories on staff motivation, it is less clear how these stories are being used, to what end they are collected, and how often they need to be shared to sustain initial levels of motivation (54).
Research and practice implications
Our findings found four key strategies (use of collaborative approach, A&F, informal leadership, and patient stories) perceived by participants to positively influence intervention sustainability. However, our research also highlighted knowledge gaps that require further research. There is a lack of rigorous evaluations on the use and effectiveness of LCs as a strategy to aid behavior change to reduce the knowledge to action gap. More research needs to be done to look at the design, components, delivery, and impact of LCs as a strategy to help with implementation and more critically, sustainability of an intervention. For A&F further research needs to be done to evaluate different approaches to the design, delivery, and dose of this intervention. We also recommend research that can unpack and try to explain theory used in A&F design and effect modifiers of A&F. Lessons from such research can help researchers and decision- makers plan, design and execute improvement interventions in a way that can be done before implementation and that can lead to sustainable outcomes and impact. Our research recommends that senior leadership needs to plan for strategies to help informal leaders to emerge and to ensure that they have the capacity and capabilities to lead intervention implementation and sustainability efforts. Patient stories have been identified as powerful strategy to translate knowledge, however evaluations are needed in relation to the use and impact of patient stories for sustainability.
Like previous research on sustainability (22) our findings illustrate the important relationship and “ripple-effect” between implementation and sustainability; where there is a causal relationship between implementation processes and outcomes, and sustainability. We found that implementation factors and decisions made for implementation were critical to facilitating or hindering contexts for sustainability. Sense making of monitoring and outcomes data was also a common mechanism at early implementation that enabled or hindered the likelihood of sustainment. Our work also aligns with and extends existing theoretical approaches for sustainability. For example, the Consolidated Framework for Sustainability presents 40 determinants that influence the sustainability of healthcare interventions, such as leadership and champions, monitoring progress over time stakeholder participation and involvement (31). Our research offers potential strategies (i.e. learning collaboratives, A&F, and patient stories) to increase the likelihood of intervention sustainability and impact. Understanding how to sustain scaled interventions, through which strategies is a novel area in sustainability research. We recommend future research that tests the effectiveness and validity of these strategies for sustainability across other scaled interventions.
In this current evaluation of two provincial wide, scaled, multi-component interventions, many of the important factors and mechanisms that had a perceived effect on sustainability were contextual factors in existence prior to implementation (e.g., leadership) or elements related to implementation (e.g., interventions designed for implementation). Future research is needed to examine how these factors have an important role to play in sustainability, not just implementation.
Resource allocation is challenging in health systems, thus it is important for implementers to understand what they ‘need to do’ vs ‘what is nice to do’ in order to create and maintain interventions that have sustainable impact. Our research has shown that a collaborative approach that includes A&F, informal leaders and shared patient stories has a perceived positive influence on sustainability; yet it remains unknown which of these strategies are a ‘need to do’ versus a ‘nice to do’ for long-term sustainability and impact. There is also a clear tension between implementation and sustainability, it is unclear for operational leaders how much effort to put into sustainability planning prior to implementation when it is unknown if an intervention will be successful or not. Nonetheless, our research emphasizes a clear relationship between implementation and sustainability; we anticipate that if SCNs can understand key components of sustainability earlier, their implementation and sustainability planning could become increasingly deliberate and efficient.
The contextual factors and mechanisms identified in this evaluation are based on the perceptions of our participants from two scaled interventions; additional research is needed to test the influence of these factors on sustainability, in situ, and among other scaled interventions. It was beyond the scope of this study to examine the sustainment of the interventions in terms of impact on clinical outcomes. To mitigate this limitation, we purposely sought out several data sources (SCN leaders, documents, including theory and existing evidence to inform the link between implementation and sustainability, participant interviews) to inform our work across all stages of the research. Our sampling of individuals within each intervention attempted to access those who could best reflect on intervention implementation and sustainability. During our Case B interviews, we learned emergently that health care aides may be a key informant role that we had not yet accessed. We subsequently attempted but were unsuccessful at recruiting individuals to participate in study interviews, and this may have negatively impacted our ability to fully characterize unique aspects of that intervention in our study.