An Idea Worth Sustaining: Evaluation of the sustainability potential of Choosing Wisely across Ontario Community Hospitals and Family Health Teams

Evidence shows that 30% of healthcare services are unnecessary and unlikely to benet patients and may even be harmful. Choosing Wisely (CW) is a physician-initiated, international campaign (with campaigns in at least 20 countries) aimed at addressing unnecessary medical care through the provision of evidence-based resources and clinical guidance. Our objectives were to evaluate the sustainability potential of CW across four Ontario community hospitals and aliated family health teams (FHT). We recruited CW implementation teams across Ontario hospitals and their aliated primary care family health teams (FHTs) to participate in a mixed-methods study to identify and respond to their CW implementation and sustainability challenges. The study involved the administration of a validated sustainability Survey at baseline, 6-and 12-months follow-up, and participation in two focus groups (at baseline and 6 months) to discuss identied successes and opportunities for improvement, and to respond to implementation and sustainability challenges through a team exercise involving the co-creation and implementation of an action plan to directly address them. our interpretation of ndings. We looked for commonalities and contradictions of themes emerging from the data, noting reections, isolating patterns and processes, and gradually building generalizations 41 . Survey scores across the three time points were used to create line graphs to show changes in sustainability scores across time. We also sent a report for each team to facilitate each team’s implementation efforts outlining their setting prole, NHS sustainability survey results over time, top successes and challenges, focus group results (action planning to address challenges), and overall key points and lessons learned.

Communication and awareness of an innovation such as CW is key to sustainability because people need to feel that what they are doing is making a difference and that the additional effort is worthwhile.
Choosing Wisely (CW) teams across Ontario community hospitals were empowered to identify and respond to implementation challenges over time to maximize the sustainability potential of their CW initiative Background It is estimated that 30% of health care services, tests, treatments and procedures are unnecessary and unlikely to bene t patients [1][2][3] , and can also lead to harm for patients, wasted resources, and longer wait times for care [3][4][5][6][7] .
Choosing Wisely (CW) is a physician-initiated, national campaign aimed at reducing overuse by addressing low value care that is unlikely to bene t patients or even cause harm, focusing on reducing overuse of care 8,9 . The goal of the campaign is to facilitate the communication between physicians and patients about unnecessary tests, treatments and procedures through the provision of evidence-based resources and clinical guidance 8 , and to help them to make informed decisions to ensure high-quality care 9 . The campaign has now spread internationally to at least 20 countries 10 including the development of recommendations to reduce unnecessary tests, treatment and procedures 11 .
The successful implementation and spread of evidence-based de-implementation innovations such as CW and their potential to improve care can only be achieved if they are sustainable [12][13] . This is particularly important for CW because despite the overwhelming worldwide commitment to its adoption, its effect on decreasing unnecessary care has been slow [14][15][16] . While this is not surprising to some extent given the wide range of CW priority areas, settings and dissemination and implementation efforts, "de-implementation" of care services (particularly when part of usual care or routine use) is not well understood but has been de ned as the "abandonment of medical practices or interventions that have been found to be ineffective and harmful" 17 . We need to identify the factors that in uence behaviour change in the context of de-implementation. Sustainability is a key implementation outcome and has been de ned as: "the degree to which an innovation continues to be used after initial efforts to secure adoption are completed" 18 or becomes a routine part of practice to support continuous care delivery 12,19−20 . Ensuring sustainability capacity is also needed to scale up innovations 21 , which can increase the impact of effective innovations through "vertical diffusion", a deliberate, systematic approach to increasing coverage, range and sustainability 22,23 .
Although sustainability is an important consideration during the implementation of innovations including those aimed at de-implementation such as CW, it is seldom considered or not considered early enough in the implementation process 24,25 . It is not surprising that up to 40% of all new programs or interventions don't last beyond the rst 1-2 years after initial funding 26,27 . Another challenge is that investigation of the long-term sustainability of innovations is lacking 13,20,24,28,29 , focusing mainly on short-term outputs, neglecting the long-term, decision-informing outcomes needed for practice and policy domains 25,30 . These de ciencies can lead to implementation failure and wasted resources 4-6,31−33 , and worsen patient outcomes and quality of care 6,24,34,35 .
Increasing "awareness" about low value care is not enough to achieve sustainable health impacts. We need to demonstrate long-term bene ts of effectiveness and uptake of innovations through behaviour and practice change, demonstrate increased patient knowledge and acceptance of these new innovations, and to show value to the system through decreased health care utilization 9,10 . The success of ongoing implementation of innovations is therefore largely dependent on their sustainability potential. Critical to this assessment are to understand the determinants of sustainability, to involve and engage all relevant knowledge users who are part of the process to implement the innovation, and to empower the implementation team to address identi ed barriers. This type of assessment has the best potential to facilitate implementation efforts of change initiatives such as CW and to support sustained improvements in health care services and patient outcomes over time 13 . The objectives of our study were to: (i) assess the sustainability potential of CW across Ontario community hospitals and their a liated primary care sites; (ii) to work with CW implementation teams to optimize their efforts; and (iii) to evaluate our overall process to advance the science of sustainability.

Design overview
Our objectives were addressed using a 12-month, multicenter, sequential mixed-methods design 36 (Fig. 1

Recruitment of CW teams
In Ontario, North York General Hospital (NYGH) was an early adopter of CW and began implementing the initiative in 2014. In 2016, the hospital received funding to spread CW to a network of Ontario community hospitals (Joint Centres for Transformative Health Care Innovation), which is a partnership between NYGH and six large community hospitals and 13 a liated primary care centres in Ontario. The goal of the Joint Centres is to improve the quality and value in health care, and to share innovative practices. Leveraging the governance structure of the Joint Centres advisory board consisting of hospital and primary care clinic and administrative leads, we used a purposive sampling strategy to identify teams that were involved in the implementation of a CW initiative (e.g., de-prescribing proton pump inhibitors [PPIs], reducing BUN/Urea lab testing) at their sites between 2017-2018. Team members could be front-line healthcare providers (i.e., nurses, pharmacists, physicians, trainees), decision-makers (i.e., chiefs, directors, managers) and researchers. We identi ed 24 potentially eligible teams from ve community hospitals (NYGH, MSH, SJHC, MGH SLRHC) and six FHTs a liated with these hospitals. We held information sessions with each team to explain the study process including participants' roles and expectations, to identify additional members of their CW implementation team, and to enrol them into the study. PHASE 1: Baseline Assessment of sustainability (T1) We used a two-step process to assess the baseline sustainability potential of each team's implementation of their CW initiative.
Step 1: NHS Sustainability Survey and Model: Sustainability survey: With the help of site administrative and clinic leads, we emailed a validated sustainability survey developed by the UK National Health Service (NHS) Institute for Innovation and Improvement Program 37 to all consenting CW implementation teams (n = 15). The NHS Sustainability survey is based on a validated model representing 10 factors across three domains that are considered important in sustaining change in healthcare 38 . Respondents are asked to provide their perceptions on each of the 10 sustainability factors by indicating the level that best t with their current implementation situation.
The goals of the model and survey are to help teams self-assess their implementation efforts and to become aware of the challenges (and successes) associated with a new practice chance such as CW. More speci cally, the survey helps reveal the strengths and weaknesses of implementation, and to predict the likelihood of sustainability. Teams were given two weeks to complete the survey with follow-up email reminders to non-responders at 2 and 4 weeks after the due date 39 . Individual survey scores were combined to derive an overall mean team score. Analysis: To identify the best potential for improvement across the 10 sustainability factors, we calculated the largest difference between actual scores and maximum potential score (a pre-speci ed maximum score built into the NHS survey) 38 ; to identify which factors contributed to successful implementation, we calculated the least difference between actual score and maximum potential score. Individual survey scores are collated, and aggregate team scores can be generated. An overall sustainability score of > 55% is considered a "reason to be optimistic" (i.e., above the threshold of what is considered a potentially sustainable intervention); and a score of < 45 suggests that some action needs to be taken to "increase the likelihood that the improvement initiative will sustain" 38 . We also investigated which sustainability factors were associated with the greatest potential for improvement (i.e., largest difference between team score and maximum potential sustainability score). Outcomes: Overall baseline mean team sustainability scores and mean scores for each of the 10 sustainability factors across the three domains of the NHS Sustainability survey.
Step 2. Action planning focus groups: Action planning focus groups: Two weeks after completing the NHS Sustainability survey, teams were invited to participate in one-hour action planning focus groups to obtain a more indepth understanding of their CW implementation. Each team was given a summary report outlining the team's overall baseline sustainability survey results (including scores and accompanying bar graphs) aimed at facilitating a discussion among team members on how to address identi ed challenges, and to select at least one challenge for which they could co-design an action plan that could be feasibly implemented within 3 months. The team survey scores were meant to start the "conversation" around identi ed sustainability challenges 38 . Focus group questions were driven by the NHS Sustainability survey scores as well as questions embedded within the NHS Sustainability model, which offers practical advice on achieving sustained use of the initiative 38 . The guide is meant to raise early awareness of sustainability challenges, and the opportunity for teams to iteratively address these challenges to optimize the new initiative's potential for impact 38 (Outcome set 1). Another goal of the focus groups was to encourage teams to co-design a plan that would directly respond to identi ed challenges revealed by the survey (Outcome set 2). The in-person focus groups were facilitated by an experienced moderator who helped to formulate action-based recommendations and strategies to improve the sustainability potential of their respective implementation efforts, focusing on the areas of opportunity as identi ed through the survey analysis. The process was aimed at raising early awareness of sustainability challenges and the opportunity for teams to iteratively address these challenges to optimize the new initiative's potential for impact. Analysis: Sessions were audio recorded and transcribed verbatim. Two reviewer pairs used a multi-step thematic analysis process 40 to independently read transcripts, create an initial list of codes, meeting to develop a codebook through consensusbased discussions, assigning codes to transcripts, discussing the main themes of the data with all team members, and deciding on how best to organize and represent data. PHASE 2: Three-month follow-up (T2) Three months after the baseline administration of the NHS sustainability survey and focus groups, we readministered the survey to all teams to assess whether the action plans designed by teams were implemented and if any new challenges have emerged that needed to be addressed. We used the same methods and procedures for assessment as at baseline. We invited teams for a second focus group, to discuss the results of the survey and how it compared with baseline data (Outcome set 1). The action plan items were discussed in the context of how they were applied in the previous three months, and if any further changes needed to be made to address new challenges (Outcome set 2). At the conclusion of these focus group each team was given a summary report outlining: (i) any changes in survey scores from baseline and accompanying bar graphs; and (ii) a description of actions that were implemented in response to identi ed challenges from the baseline assessment. PHASE 3: Final follow-up (T3) Three months after the second administration of the NHS sustainability survey, we administered it at the conclusion of the study to identify any changes in sustainability scores over time. In addition, we included an evaluation form to assess the impact and value of our study and overall process to advance the science of sustainability (Appendix X). Survey questions: Survey questions consisted of 5-point Likert type questions (where 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree) and open-ended questions. Outcomes: Participants were asked to rate their agreement with statements about the CW sustainability study and its speci c features and to provide any additional comments about the value and usefulness of the: (i) information sessions; (ii) the NHS Sustainability survey including its usability; (iii) focus group sessions (to understand the survey results, to identify the team's sustainability challenges, and if they helped to formulate an action plan to address challenges); (iv) action plan report (readability and organization, whether the team used the information in the report to make changes to their CW implementation process); and (v) to provide an overall assessment of the feasibility of participating in the study (i.e., completion of three surveys and participation in two focus groups), satisfaction; and whether the information they received throughout the study has increased their knowledge about the concept of sustainability. Analysis: Descriptive statistics for quantitative data (frequencies, means) and content analysis 41 for qualitative data.
Qualitative data was also triangulated with quantitative assessments to support data interpretations. We used a meta-matrix method 42,43 to triangulate survey and focus group data to facilitate our interpretation of ndings. We looked for commonalities and contradictions of themes emerging from the data, noting re ections, isolating patterns and processes, and gradually building generalizations 41 . Survey scores across the three time points were used to create line graphs to show changes in sustainability scores across time. We also sent a report for each team to facilitate each team's implementation efforts outlining their setting pro le, NHS sustainability survey results over time, top successes and challenges, focus group results (action planning to address challenges), and overall key points and lessons learned.

Site and participant characteristics
We conducted 12 information sessions with 15 potentially eligible teams across four hospitals and six FHTs. Of these, all 15 teams representing 91 individuals consented to participate. The site and participant characteristics of hospital and primary care teams are shown in Tables 1 and 2, respectively. A total of 173 surveys were completed across three time points: baseline (n = 67), rst follow-up (n = 59) and nal follow-up (n = 47). Twenty-nine focus groups were held in total; 15 at rst follow-up and 14 at nal follow-up. Teams consisted of different combinations of health care professionals, chiefs/directors/managers, and clinic and administrative staff: clinicians (nurses, physicians, pharmacists), trainees (residents, pharmacy students) and health information data managers.  Cardiologist; Head of non-invasive cardiac lab and echo lab (n = 1)  EMR system administrator (n = 1) NHS sustainability survey Table 3 shows the mean NHS Sustainability survey team scores over time (T1, T2, T3) across hospital and primary care sites and their respective CW priority areas (Figs. 2-5). The mean baseline team scores across all sites and teams ranged 65-87%, which is above the threshold of what is considered a potentially sustainable innovation (i.e., ≥ 55%) 38 . Survey results are organized according to the CW priority areas investigated by hospital and primary care family health teams (FHTs): , 3-month follow-up (n = 24) and nal follow-up (n = 22). The baseline mean NHS sustainability scores ranged 53-74% and increased to a range of 66-80% at 6-months and to 70-83% at 12-months (Fig. 4). The mean sustainability scores increased from baseline to 12-month follow-up for all six teams (mean increase 10%; range 5-17%). Hospital teams: Three hospital teams completed the NHS sustainability survey at baseline (n = 17), 3-month follow-up (n = 16) and nal follow-up (n = 10). The baseline mean NHS sustainability scores ranged 59-72% and increased at rst follow-up to 64-69% (two teams) while the third team showed a 5% decrease from 72-67%. At 12months follow-up, the mean sustainability scores overall increased from baseline for two teams (mean increase 6%; range 2-10%); and decreased for the third team by 4% (Fig. 3). This decrease may in part be explained by a few challenges that we identi ed from triangulating survey with focus group data: 1) incomplete and wide variability in participation by team members across the three time points due to staff and student turnover; 2) the program lead went on maternity leave (T2) and therefore was not able to complete the nal survey; 3) the team has not implemented CW hospital wide. At the time of T1, the team started a pilot of the initiative only on their alternative level of care (ALC) oor. The team acknowledged that their current processes needed to be changed to ensure successful implementation and long-term sustainability once the initiative is rolled out hospital wide.
BUN/Urea lab testing: Hospital teams: Three hospital teams that implemented the BUN/Urea CW priority completed the NHS sustainability survey at baseline (n = 11), rst follow-up (n = 12) and nal follow-up (n = 9). Their baseline mean NHS sustainability scores ranged 61-89% and increased at rst follow-up for two teams (range 72-85%) while decreasing by 3% for the third team from 61-58%. At 12-months of follow-up, the mean sustainability scores increased from baseline for all three teams (mean increase 13%; range 2-27%) (Fig. 4).
Pre-Op testing: Hospital teams: Two hospital teams that implemented the Pre-Op testing priority completed the NHS sustainability survey at T1 (n = 11), T2 (n = 12) and T3 (n = 9). Their baseline mean NHS sustainability scores ranged 82-92%. One of the teams showed a steady increase in sustainability scores over time with a 5% increase from T1 to T3 (Fig. 5) while the second team showed a 25% decrease in scores from T1 (82%) to T2 (68%) to T3 (57%). This large decrease may be attributed to several important challenges that we identi ed by triangulating survey with focus group data: 1) the implementation of their CW initiative did not occur until T2 ( rst follow-up); 2) implementation was extremely complicated at this site because they have a total of 55 surgeon's o ces not all of which are located at the hospital, and the hospital was in the midst of a merger with two other hospitals; 3) the site is not fully electronic/automated; 4) the Pre-Op patient care manager went on maternity leave midway through the study (T2), which affected the participation of the team, since this person acted as the point person to send out surveys and coordinate focus groups; and 5) the team's high baseline sustainability team score (82%) may be explained by the team's "hopefulness" for the initiative, since they have not yet implemented it at the start of the sustainability study (T1). A clinician member of the team indicated that the much lower scores observed in the later parts of the study may have been due to the burden of the CW initiative not being felt by the team until they launched it (T2), the high learning curve of the initiative, more work for front line staff, and not having a fully electronic/automated system to accommodate CW processes.
IP Echo: Hospital team: One hospital team implemented the IP Echo CW priority. The team had two members who completed the NHS sustainability survey at baseline (mean sustainability score range 73-85%) and rst focus group but did not complete T2 and T3 surveys or nal focus group.
Top challenges and success factors Figure 6 shows that the largest proportion of implementation teams perceived t with existing processes and work ows as the success factor for sustainability regardless of the CW priority areas that we investigated (i.e., Hospital/FHT PPI de-prescribing, hospital BUN/Urea testing and Pre-Op testing) followed by leadership support and optimized team communication. In terms of challenges, the largest proportion of teams indicated (regardless of CW priority area) that lack of awareness and buy-in, lack of leadership engagement or a champion and lack of t with existing work ow and culture (Fig. 7). There was a 78% overlap in challenges between FHT and hospital teams that implemented the PPI de-prescribing priority. Among the six FHTs, challenges identi ed by the majority (50% or more) of teams were lack of leadership engagement or a champion, frequent staff turnover, volume of QI initiatives, and the time consuming and resource intensive nature of PPI de-prescribing. Among hospital PPI teams, two thirds found lack of awareness and buy-in and training as top challenges.

Qualitative Focus Groups
Analysis of 29 focus groups revealed three themes related to sustainability facilitators (Appendix A) and four themes related to challenges (Appendix B). All teams identi ed at least one challenge for which they co-designed and implemented an action plan to maximize the sustainability potential of their CW priority area.

Sustainability facilitators
Fit with existing processes and work ows Four primary care teams indicated that CW is an intuitive campaign because it's built into the system so even if teams change, the project can continue. Both primary care and hospital teams indicated that PPI de-prescribing is well supported by CW because it includes tools that support optimized implementation (e.g., the ability to ag which patients should be reviewed for PPI de-prescribing, algorithms and tapering schedules, and electronic order sets that include a default order for a pharmacist's consult for a full medication review). For Pre-Op testing, hospital teams indicated that CW is a straightforward, standardized approach, and can increase the e ciency of clinical work ows (i.e., less paperwork for administrative staff and more exibility in nurse scheduling). BUN/Urea teams perceived ease of implementation due to its ability to be implemented almost anywhere, and the relative straightforwardness of the process (i.e., since the test can be removed from the post-surgery order set, it can eliminate "automatic" or "default" ordering).

Leadership support
Three primary care teams identi ed leadership support for successful PPI de-prescribing likely attributed to supportive clinical and senior leaders and staff involvement and that the initiative was prioritized by clinical leadership as a patient safety issue. Since pharmacists are central to the PPI de-prescribing process, this also freed up the involvement of family physicians, although having clinical champions was perceived as important for success. For the two hospital PPI teams, support from senior leadership on an organizational level was perceived to be the reason for success. Two Pre-Op testing hospital teams indicated that change could not have been possible without clinical lead champions and Joint Centres support, and a BUN/Urea team attributed success to corporate level leadership.

Optimized team communication
One FHT has taken opportunities to present data on their de-prescribing experience, which fostered an environment to encourage staff involvement. A hospital PPI team experienced enhanced team collaboration and communication because of regular communications between pharmacists and providers to discuss PPI de-prescribing and reassessment of therapy were facilitated by the CW initiative. They also found that available CW publications helped support changes in the process (de-prescribing algorithm, pamphlets for patients and pharmacists). Two BUN/Urea teams created CW posters, which has been effective in creating/raising awareness among ICU staff, which in turn has facilitated reduced ordering of these tests.

Sustainability challenges and actions to address them
Lack of awareness test, which made it more di cult to "sell" the initiative. To overcome this, some teams will perform an interrupted time series analysis to evaluate the impact of their implementation. Another team felt that poor uptake stemmed from nursing staff not understanding the reason for reducing BUN/Urea testing because they are not adequately engaged in the initiative. To overcome this challenge, the team engaged in conversations with and listened to frontline staff about the importance of the initiative, engaged in education and presentations to gain support and consensus. Another BUN/Urea team found it di cult to change engrained behaviour patterns and habits of staff, who may just "automatically select blood work on the standard order sets". To acknowledge this challenge, the team conducted an "audit and feedback" exercise to review blood work volumes of various clinics and departments and shared these results. The team's clinical lead has reached out to physicians to engage in a conversation about the appropriateness of BUN ordering, and the team's lab manager also reached out to clinic teams to support behaviour change efforts. Lack of t with existing work ows and culture

Lack of leadership engagement or a champion
One primary care team felt that the PPI de-prescribing initiative is not yet part of the organizational work culture, and they would like it to become a habit just like other process such as checking blood pressure. To do this, they are carrying out a number of interventions including education, communication, monitoring with audits and feedback.
The FHT is also continuing to support a research project (which could be done by residents) to look at aggregate PPI data at different sites to provide more evidence for physicians on the bene ts of the initiative. They also noted that non-academic primary care sites will struggle to incorporate CW into their work culture because they don't have the academic work culture, which tends to promote patient safety, include a pharmacist and data manager, and have a built-in system for staff training and improving processes -these are features that make FHTs amenable to initiatives such as CW.
Evaluation of our process 45 of the 47 individuals across 15 teams who participated in PHASE 3 of our study (T3) completed the evaluation survey (response rate 96%). Appendix C shows the dispersion of mean scores on the evaluation survey of participant perceptions of our study overall, the information sessions, the NHS sustainability survey, the action planning focus groups, and the action plan reports. Overall perception of the Sustainability study: Respondents agreed that their participation in the sustainability was feasible, helped strengthen the implementation of their CW priority area, increased their knowledge about the concept of sustainability, and were overall satis ed with the study (mean score range 4.0-4.3 out of 5). Although teams agreed that they would need to continue meetings to sustain or maintain their current activities (mean 4.0), it was perceived less likely that their teams would meet regularly about the CW priority (mean 3.9). Another FHT member described the implementation of an innovation such as Choosing Wisely as challenging in a "climate of multiple competing demands" that requires a "true collaborative approach to make successful". Information sessions: Teams agreed that the information sessions were effective for introducing the CW sustainability sub-study, helped them to understand the objectives of the study and the NHS sustainability survey, and found the information session overall useful (mean score range 4.1-4.3). NHS Sustainability Survey: Respondents perceived the NHS Sustainability survey as easy to complete and understand, took a reasonable amount of time, and helped to learn about the sustainability study (mean score range 4.0-4.3). A small sub-set of respondents (7%) indicated that the NHS Sustainability Survey was "in exible", and not re ective of their actual circumstances, which made it di cult to answer the questions. Action planning Focus Groups: Respondents agreed that the action planning focus groups were overall helpful, well organized with effective facilitators and helped team members understand the survey results and to identify sustainability challenges (mean score range 4.0-4.3). There was slightly less agreement by respondents about the focus groups helping teams to formulate an action plan to address challenges (mean score 3.9). Team Reports: Action plan Reports summarized each team's sustainability scores, top challenges and successes, and provided an operationalizable summary of the plan and processes cocreated by the team to address challenges. Respondents agreed that these reports were easy to read and well organized (mean score 4.1). However, they had less agreement for using these reports to make changes to their CW implementation process, whether teams perceived that these changes led to improvements in their implementation; and whether three months was a long enough period to implement their co-designed action plans (mean score range 3.5-3.6).

Discussion
Overall, our study showed that all 15 implementation teams across ve hospitals and six a liated primary care sites achieved a baseline sustainability team score that was well above the threshold of what is considered a potentially sustainable innovation (i.e., ≥55%) 38 . We observed steady improvements in sustainability scores over three time points across all teams, which is a strong indicator that CW is sustainable in hospital and primary care settings for the CW priority areas that were investigated (PPI de-prescribing, BUN/Urea testing and Pre-Op testing).
Action planning focus group sessions revealed that regardless of site or CW priority area, facilitators that were common across all teams were Fit with existing processes and work ows, Leadership support, and Optimized team communication (Fig. 8); common challenges were Lack of awareness and buy-in, Lack of leadership engagement or a champion, and Lack of t with existing work ow and culture (Fig. 9). All teams identi ed at least one challenge for which they co-designed and implemented an action plan to maximize the sustainability potential of their CW priority area.
Actions taken by teams to address Lack of awareness and buy-in included sharing information via newsletters, meetings and emails; creating a simple, one-page infographic; incorporating PPI de-prescription as part of resident training; holding educational events and meetings with staff; making posters and handouts available for patients; and to engage in publications, presentations and posters. To increase buy-in for patients, one FHT sent a mass email about PPIs which led to patient visits about de-prescribing. To encourage buy-in for staff, one clinic encouraged those who regularly engage in CW activities to meet with those who do not; another team's clinic lead presented at operations committees and grand rounds; a team undertook a trial to evaluate the impact of their implementation; while others engaged in conversations with frontline staff, provided education and presentations to gain support and consensus; performed an "audit and feedback" exercise, and reached out to clinicians about the appropriateness of lab tests and to support behaviour change efforts. Actions taken by teams to address Lack of leadership engagement or a champion included trying to rotate clinical leadership; contacting and learning from sites which had experience and success with the initiative's implementation; reaching out via email and conversations; engaging the QI specialist to gather data to help support engagement efforts; and to set up an ongoing dialogue to engage site leadership prior to the nal changes in order sets. Actions taken by teams to address Lack of t with existing work ow and culture included making the initiative become a habit (similar to taking blood pressure) by carrying out a number of interventions including education, communication, and monitoring with audits and feedback; supporting research projects, which could be done by residents; and to think about how to provide CW to non-academic sites.
The sustainability scores at baseline were about 5-22% lower for teams that implemented the PPI de-prescribing initiative (regardless of setting) compared with other CW initiatives that were implemented in hospital (BUN/Urea, Pre-Op, IP Echo). Our focus group data supports this observation as the majority of challenges (apart from the three that were common to all CW priorities) were clustered around the FHT and Hospital PPI priority areas (78% overlap). Figure 7 highlights this clustering effect, which may in part be explained by the level of complexity required to implement PPI de-prescribing compared with other CW initiatives that may be less complicated (e.g., removing lab or imaging tests from order sets). PPI de-prescribing requires behaviour change for implementation (both from the provider and patient perspective). For example, the prescription PPIs is in part driven by patients, who may not understand the bene ts of stopping or demand a re-prescription from their provider if their gastrointestinal (GI) symptoms return. Patient perceptions can in turn in uence provider prescription behaviours. From the provider's perspective, complexity might stem from feeling pressured by patients to re-prescribe PPIs if their GI symptoms return, not being aware of the evidence (or lack of buy-in) for the de-prescribing effort, and not seeing the immediate bene ts of stopping PPIs. These complexities highlight that we need to better understand not only the mechanisms of the implementation process (i.e., PPI de-prescribing), but also the various knowledge user perceptions, and to design interventions that best match or mitigate identi ed challenges. It also highlights the importance of engaging patients to contribute to the optimization of implementation efforts.
Implementation of complex interventions such as CW is important to achieve meaningful health impacts, but this is challenging. Several focus group participants indicated that implementation of CW and required behaviour changes takes time, particularly at the early stages (e.g., to assess what needs to change to support the new initiative, to set up systems). One team took action to anticipate implementation problems by taking extra time to gure out the processes prior to implementation and felt that it was important to do this to ensure long-term sustainability of their initiative. We also observed that a single change during implementation can have both intentional and unintentional consequences. For example, in the PPI de-prescribing initiative, family physicians recognized that patients have a much larger in uence on prescribing decisions than expected. Patients are driving prescription decisions either by pressuring family physicians to be put back on PPIs or by "going around the system" to obtain a prescription through a GI specialist. Unintentional consequences are inevitable when implementing complex interventions, but this underscores the importance of understanding the determinants of usual practice and how the new way of doing things (i.e., CW) will in uence improvement efforts. Another suggestion was to "not reinvent the wheel" by learning from others who have already implemented CW so that teams don't have to start from scratch. Suggestions were to develop an information linking mechanism or platform to share experiences such as through an online portal, and to have knowledge brokers or implementation facilitators to support implementation efforts. Communication was identi ed by several participants as a "key to making it [CW] work", which is in line with one of the common challenges that we identi ed across all sites and CW priority areas: "lack of awareness". In particular, participants suggested that the modes and purpose of communication are varied but important to create awareness (of the CW initiative and its impact), buy-in, and to have information delivered by a credible source (champions) in a balanced way (right amount, right time), and to "keep it going until it becomes routine or established practice behaviour". Given that "people are busy so it's easy to forget", participants also indicated that communication needs to be "creative" to be effective. For example, to come up with a simple and easy way to draw attention to the CW initiative for staff, as did a BUN/Urea team that placed an infographic on top of their nursing workstation.
We found that several aspects of our study were essential for measuring and encouraging sustainability that can be considered by others adopting an innovation such as the CW campaign: Information sessions. We held information sessions with all participating teams where we explained the study, highlighting the importance of sustainability and outlining expectations for study participation. These sessions helped solidify teams (not all team members had previously worked together) and taught them how to complete the NHS Sustainability Survey. Action planning. The NHS Sustainability Model does not include a process for creating dialogue among implementation team members, although this is encouraged. In response, we held formal focus groups with teams. These helped focus discussions (which may not otherwise have happed on their own) and led to the majority of teams to formulate an action plan in response to at least one sustainability challenge. Repeat measurements of sustainability. Bringing teams together in focus groups and asking teams to repeat the sustainability survey over 3 time points not only provided an opportunity to identify and address sustainability barriers but also became a reinforcing activity for teams to identify and respond to their implementation challenges. Using experienced facilitators. Experienced facilitators helped participants understand the underlying concepts of sustainability, the NHS sustainability model and the survey results, stimulated discussion, helped teams identify the challenges that could feasibly be addressed, and shared examples and experiences of other teams with similar processes or challenges. Generalizability of our ndings: As part of our strategy for sustainability, we embedded a step to facilitate the maintenance of each team's CW initiatives. Each team member received a team report along with a conceptual Implementation and Sustainability Guide (Appendix D) to help further support their current and/or new CW implementation efforts. Our future work will involve validating this guide so it can be applied by other implementation teams and sites.

Strengths and Limitations
Our study had many strengths. First, to our knowledge, we are the rst to evaluate the sustainability potential of an implemented Choosing Wisely campaign. Most CW evaluations are focused on measuring the impact of the campaign and whether their implementation has been associated with reductions in low value care 9,44,45 , while others have assessed awareness and perceptions of the campaign by providers 45 . We contributed to the current limited knowledge of ensuring the sustainability of innovations by evaluating the sustainability potential of the CW campaign across ve Ontario hospitals and FHTs. Sustainability is seldom considered in implementation efforts 24,25 or considered early enough to be able to address its potential challenges and barriers 25,30 . In fact, there is a signi cant evidence gap in investigating the sustainability of effective interventions 24 , and only a few examples illustrate the application of the NHS sustainability model/framework 20,21 . Our work contributes to advancing the limited knowledge of sustainability and its methods and how implementation teams can be empowered to in uence their implementation efforts. Although the NHS Sustainability Model includes probing questions to encourage the dialogue among implementation team members, there is no guidance on how to achieve this. We built on this work by creating a strategy to ensure full engagement of teams through action planning focus groups, using discussion questions informed by the sustainability surveys. Our involvement (i.e., mobilizing teams and providing opportunities for a solutions-focused discussions) may have strengthened their implementation by empowering teams not only to identify and respond to challenges but also to recognize and leverage their strengths. Most teams who engaged in our process have adopted CW priorities into their practice and have made it part of their routine care to support care delivery -this is the very de nition of sustainability 12,19,20 . Third, bringing teams together in focus groups (at 2 time points) and completing the sustainability survey (at 3 time points) became a reinforcing activity for teams to identify and respond to their implementation challenges. In fact, systematically bringing teams together and facilitating discussions may have become an intervention in itself. This highlights a need for a formalized process to engage implementation teams, so they have a forum to discuss and work through their challenges. The question remains, how much involvement do teams need to optimize implementation? It's important to understand implementation teams and their context and to tailor strategies according to their variable needs. Fourth, using experienced facilitators during our qualitative sessions helped participants to understand the underlying concepts of sustainability, and the survey results, to stimulate discussion and prompt ideas, and to encourage teams to apply their new knowledge into practice. Evidence shows that facilitation is becoming an important aspect of implementation science, and has potential to support innovations for optimized impact 20,47 . Lastly, ensuring sustainability capacity is needed to scale up innovations 21 . One of our outputs was a conceptual sustainability and scalability guide, which after further validation could increase the potential for any future CW campaigns to be successfully scaled up at other hospital and primary care settings in Ontario.
Our study had some limitations. First, not all teams were at the same implementation stage of their CW initiative, which may have reduced the reliability of our trend analysis; mid/late implementers had different experiences from early implementers. However, we used focus group data to explain divergent trends including the decreased implementation scores we observed for two of the 15 teams. Second, we observed variable rates of attrition and participation both within and across teams, which may have diminished the representativeness of our teams.
However, frequent staff changeover (e.g., revolving residents and students, maternity leaves) are a normal part of clinic team operations, so our participation rates are a re ection of that reality. Not surprisingly, hospital teams had a higher attrition rate than FHTs (39% vs. 15%) in part because the nine hospital teams had variable implementation schedules whereas all six FHTs had fully implemented their CW priority (PPI de-prescribing) at the start of our study.
Additionally, the governance and operational structure of FHTs were more streamlined; all six FHT teams were governed by one clinic lead and focused on a single CW priority area (i.e., PPI de-prescribing) compared with the nine hospital teams that implemented four different CW priorities (PPI, BUN/Urea, Pre-Op, and IP Echo) with each hospital having their own clinic/administrative leads. Lastly, it is not possible for us to be certain that any improvement observed was due to our study (which could be considered an intervention in itself), or due to any existing supports or other quality improvement efforts in place at the time of our study. However, we made every effort to document each team's implementation processes and work ows to identify these. In fact, we encouraged teams to implement changes that would allow embedding into their existing processes and other QI initiatives.

Conclusions
Evaluating the sustainability potential of an innovation such as Choosing Wisely is critical to ensuring that they have the best potential for impact. Our work highlights that implementation teams can be empowered to in uence their implementation efforts, and to realize positive outcomes for their healthcare services and patients. CW teams gained increased knowledge about the importance of sustainability and were given an opportunity to co-create an action plan to address their challenges. Providing opportunities for a solutions-focused discussion as part of implementation empowers teams to identify and respond to challenges as well as to recognize and leverage their strengths -these can lead to sustained improvements in health care services and patient outcomes over time.
48. Cranley LA, Cummings GG, Profetto-McGrath J, Toth F, Estabrooks CA. Facilitation roles and characteristics associated with research use by healthcare professionals: a scoping review. 2017;7:e014384.  Proportion of sustainability challenges identi ed by sites and CW priority areas Common and unique success factors across sites and CW priority areas Common challenges across sites and Choosing Wisely priority areas

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.