Implementation and Sustainability of an Enhanced Pulmonary Rehabilitation Program in a Single Centre

Background: Pulmonary rehabilitation (PR) has major benets for patients with chronic obstructive pulmonary disease (COPD). The availability of PR in Canada is low. To facilitate implementation, a standardized and enhanced PR Program has been developed with a self-management education intervention that has been validated and shown to be effective. The objectives of our study were to assess the implementation of this program into a single site; determine the sustainability 18 months after implementation; and to identify the satisfaction with, facilitators of and barriers to implementation and sustainability of the program as perceived by patients with COPD and HCPs. Methods: We conducted a prospective pre-post study which consisted of two phases: 1) the implementation phase (rst six months after implementation) and 2) the sustainability phase (18 months after implementation). Guided by the RE-AIM framework, outcomes including: Reach (number of patients enrolled and demographics), Effectiveness (Change in patient outcomes), Adoption (HCPs’ characteristics), Implementation (Program delity) and Maintenance (Patient outcomes at 3-month follow-up). Results: Reach: Twenty-six patients were included for both phases (two different samples). Effectiveness: Clinically important improvements in patient outcomes were found for functional exercise capacity, knowledge, functional status and self-ecacy in both phases of the study. Adoption: All HCPs involved in PR (N=8) agreed to participate and used the program in both phases of the study. Implementation: Fidelity for the group education sessions ranged from 76-95% in the implementation phase and from 82-88% in the sustainability phase. Maintenance (implementation phase): Clinical important improvements at 3-month follow-up were found in knowledge, self-ecacy for exercise and walking. Patients and HCPs were highly satised with the program. Lack of time was reported as the most common barrier and having longstanding procedures as the most common facilitator by HCPs to implementation and sustainability of the program. Conclusions: The enhanced PR program was accepted by the patients and HCPs and could be reliably implemented and maintained at a single expert center. The program provided clinical benets for patients in terms of functional capacity, knowledge and self-ecacy for walking. These ndings will guide planning for wide scale dissemination and implementation of the program. This study is one of few studies regarding not only implementation but also sustainability in pulmonary rehabilitation (PR) which is an essential step for further research on PR. months with high program delity of the group education sessions and most of the exercise recommendations. Moreover, the HCPs were committed to further sustainment of the enhanced PR program. Our evaluation also showed that patients had clinical important improvements in functional capacity, knowledge, functional status and self-ecacy outcomes post-program and at 3-month follow-up. These improvements are in line with the body of evidence on the benets of PR for individuals with COPD 1,24−27 .

PR has been shown to be the most effective management strategy to improve shortness of breath, exercise tolerance and health related quality of life 5,6 in individuals with COPD while also having the potential to reduce exacerbations requiring hospital admissions 9 .
Despite the major bene ts in people with COPD 1,8,10 , the availability of PR programs in Canada is low 5,11,12 . In addition, in Canada, there is currently a lack of agreed upon evidence-based standards for PR and a high degree of heterogeneity between programs. A survey of PR programs published in 2015 identi ed substantial variation between the 129 existing Canadian programs 12 . The duration ranged from 1 week to > 12 weeks, and exercise type and intensity, as well as educational topics delivered, varied widely across programs 12 .
Low program availability and quality have also been identi ed in other countries. In fact, recently, the American Thoracic and European Respiratory Societies (ATS/ERS) made recommendations to increase implementation and delivery of PR worldwide which included improving quality of PR programs and increasing awareness and knowledge of PR amongst healthcare professionals (HCPs), taxpayers and patients 11,13 .
To ensure the quality of care in PR programs in Canada and facilitate implementation, in 2016 a group of clinicians and researchers from across Canada developed a standardized and enhanced PR program with a self-management education intervention 14 that has been validated and shown to be effective 15 . This program has recently been endorsed by the Canadian Thoracic Society and includes guidelines on patients' referral and assessments, exercise type, duration and intensity, educational topics and delivery as well as long-term follow-up. It incorporates an established self-management program adapted from the "Living Well with COPD" (LWWCOPD) program [16][17][18] to create a comprehensive, multi-component PR program for individuals with COPD and provide guidance on self-management delivery according to principle of learning and cognition to modify patient's behavior. A randomized controlled trial conducted by our research team 15 compared the enhanced PR program with a traditional PR program and showed that the enhanced PR program had similar improvements in physical activity, self-e cacy, and health outcomes as the traditional PR programs 15 . However, the enhanced program had the added bene t of reducing healthcare utilization 15 .
The resources available within the enhanced PR program enable HCPs to improve current programs by using an evidence-based approach and can also be used to implement a new program in sites where PR is not yet available. Before the enhanced PR program can be implemented on a wider scale in Canada, the different phases and aspects of the implementation process of the program need to be well understood and evaluated. The RE-AIM framework 19 (Reach, Effectiveness, Adoption, Implementation and Maintenance) is a model used to translate research evidence into practice and help plan programs and improve their chances of working in "real-world" settings. It also allows programs to be evaluated on the individual and organization levels on multiple domains. We conducted a study using the RE-AIM framework as a guide in order to: 1. Assess the implementation of the enhanced PR program into a single site; 2. Determine the sustainability of the implementation of the enhanced PR program 18 months after implementation; 3. Identify the satisfaction with, facilitators of and barriers to implementation and sustainability of the program as perceived by patients with COPD and HCPs in PR.
The lessons learned from this real-world evidence project can be used by other centers that intend to implement evidence-based PR programs in their sites.

Design
This single-site study has a prospective pre-post design using the RE-AIM framework 19 to evaluate the implementation and sustainability of the enhanced PR program. The study consisted of two phases: 1) the implementation phase ( rst six months immediately after implementation) and 2) the sustainability phase (18 months after implementation). Both quantitative measures and open-ended survey-questions were used.

Participants
The program implementation was assessed on individual (patients) and organization levels (PR program/HCPs). The inclusion criteria were: Patients enrolled in the enhanced PR program, with a diagnosis of COPD con rmed by post-bronchodilator FEV1/FVC ratio of less than 0.7 1 .
HCPs involved in the PR program were invited to participate. This included individuals licensed in nursing, respiratory therapy, physiotherapy, occupation therapy, social work and nutrition.
Exclusion criteria included: patients with a diagnosis other than COPD, with cognitive impairments who were unable to accurately complete questionnaires, and who could not understand either English or French.

Study site and recruitment
The study took place at the PR program of the Montreal Chest Institute (MCI) of the McGill University Health Centre, in Montreal, Canada. The existing outpatient PR program included both weekly group education sessions, as well as supervised exercise 3 times per week, for 6 weeks. No changes were made to the timing or duration of the program. The main changes made in the enhanced PR program related to the education material used as well as the format for delivery of education sessions. The previous educational sessions and materials had been developed by PR staff and contained a mixture of didactic and selfmanagement focused education styles, the sources were not standardized (e.g. they do not all refer to the LWCOPD program or care guidelines), and their quality and content were dependent on the professional expertise. The enhanced PR program contains 12 essential education topics including slides and patient handouts which were designed by a team of content experts and they encouraged patient participation and principles of self-management education. HCPs were asked to refer to the reference guide for self-management delivery and communication with individual and group patient sessions. All the new material contains standardized references to the LWWCOPD (log in to www.livingwellwithcopd.com, section "Rehabilitation" for details of education sessions and materials).
For the implementation phase, patients were recruited from the rst three cycles of PR (6 weeks each) after implementation of the enhanced PR program (August 2017 until February 2018). Patients who were enrolled in the program 18 months after implementation (two cycles -March 2019 until June 2019) were included in the sustainability phase of the study. All HCPs working in the PR program during the implementation and sustainability phases were invited to participate and included when they provided informed consent.

The enhanced PR program
The enhanced PR program is based on current evidence and international guideline recommendations for PR with selfmanagement education based on the LWWCOPD program. The LWWCOPD is effective in promoting behaviour change, improving quality of life, and reducing hospital admissions in individuals with COPD [16][17][18] . More information on the enhanced PR program is shown in the Additional le 1.

Implementation strategy
Before implementation, all HCPs involved in the PR program at the MCI received an initial one-day group training with time set aside for questions and discussion regarding the implementation plan. During the training, the program content, goals and motivational communication techniques were presented and discussed. All HCPs involved in the program were shown how to access the resources of the program (e.g. reference guides, assessment forms as well as facilitator notes and slides for group education sessions). Feedback from HCP's involved in the program was incorporated into the nal implementation plan. Members of the study team were available throughout the initial phase to answer questions and assist PR staff as required. One of the main adaptations done by the HCPs was to schedule the 12 essential topics of the CPRP during the available 10 group education sessions (e.g. some group education sessions would need to include 2 topics).

Measures
The RE-AIM framework 19 guided the outcomes for both objective 1 (implementation phase) and objective 2 (sustainability phase). The domains of the RE-AIM Framework are depicted in Table 1. Objective 3 pertained to both phases of the study. The impact of an intervention on important outcomes, including potential negative effects, quality of life, and economic outcomes.

Individual level
Clinical outcomes of patients pre and post program

Adoption
The absolute number, proportion, and representativeness of settings and intervention agents who are willing to initiate a program.

Organization level
The proportion of HCPs following the enhanced PR program out of the total HCPs involved in PR at the study site and their characteristics Implementation At the setting level, implementation refers to the intervention agents' delity to the various elements of an intervention's protocol. This includes consistency of delivery as intended and the time and cost of the intervention.

Organization level
Fidelity on content and delivery of the group education sessions by the HCPs involved

Maintenance
At the individual level, maintenance has been de ned as the long-term effects of a program on outcomes after 6 or more months after the most recent intervention contact.

Individual level
Clinical outcomes of patients at 3-month follow-up The extent to which a program or policy becomes institutionalized or part of the routine organizational practices and policies.

Organization level
Outcomes on domains Reach, Effectiveness, Adoption and Implementation as described above within new PR cycles 18 months after initial implementation of the program.
available from www.re-aim.org

Implementation phase
Reach: Number of individuals referred for PR, number of patients with COPD participating in the enhanced PR program within the study period, reasons for not participating and characteristics of the included individuals.
Effectiveness: Clinical outcomes pre and post program (functional exercise capacity, health status, patient report of di culty or ease in daily activities, knowledge and self-e cacy) ( Table 2).  Implementation: Fidelity of the group education sessions using a checklist developed by investigators (Additional le 2) that evaluated the content (4 or 5 items) and delivery (3 items) of the education sessions. The components of this checklist were scored on a 7-point Likert scale ranging from (1) Not addressed to (7) All aspects covered clearly and in depth. One of the researchers attended a sample of the group education sessions to score.
Maintenance: Clinical outcomes (knowledge, self-e cacy and the patient report of ease or di culty in daily activities) at 3-month follow-up.

Sustainability phase
As the implementation phase was a learning process, some outcomes were changed or added for the sustainability phase. The outcomes for the Reach and Effectiveness domains were the same as the implementation phase. Changes in staff were tracked for the domain Adoption.
For the implementation domain, the same delity checklist was used to assess the group education sessions. In addition, the exercise sessions were assessed for delity. The duration, number of sessions, home exercise, types of exercise, monitoring and the use of strategies to determine exercise intensity were compared to the programs' recommendations described in the Additional le 1. These data were extracted from the patients' medical records. To understand how the HCPs were dealing with the enhanced PR program 18 months after implementation, one of the investigators observed and communicated with the all dedicated HCPs in their weekly meetings during the study period. At the group education sessions when one of the investigators was present, she observed and communicated with patients as well. This type of observation is called observer-as-participant by Gold (1958) 20 . Notes and ideas were put in a digital logbook.
As all outcomes above were assessed 18 months after implementation, further outcomes for the domain "Maintenance" would be redundant and therefore were not included in this phase.
Satisfaction with, facilitators of and barriers to implementing and sustaining the enhanced PR program For objective 3, patients and HCPs were asked to ll out surveys in both the implementation and the sustainability phases Patients were asked about their satisfaction with the program components and potential barriers for participation. HCPs were asked about their satisfaction with the content and introduction of the program and potential barriers for implementation and maintenance of the program.
To further assess the barriers and facilitators of the implementation and maintenance as perceived by the HCPs in the sustainability phase, HCPs were asked to complete an adjusted version of the Determinants of Implementation Behavior Questionnaire (DIBQ) which is based on the Theoretical Domains Framework (TDF) 21,22 .

Analysis
Outcomes on reach, adoption and implementation were analyzed descriptively. T-tests for patient characteristics were performed to identify statistical differences in participants that completed and participants that did not complete the program in the implementation phase. This was not done for the sustainability phase as there was only one participant that did not complete the program. From the delity scores of the group education sessions, a percentage was calculated for each group education session, with 'good' delity ≥ 5.
Changes in patient outcomes were calculated. Because of low sample size no statistical tests were performed. Therefore, the number and percentage of participants that improved more or equal to the minimal clinical important difference (MCID) were calculated. This data was included in the additional les. The MCIDs used were based on the literature as depicted in Table 2

Results
REACH: Implementation and Sustainability phases Figure 1 and 2 present the patient referral and participant inclusion for the implementation and the sustainability phases respectively. In the implementation phase, 54 (70%) patients out of 77 referred patients agreed to participate in a PR program. In the sustainability phase, 24 (75%) out of 32 referred patients agreed to participate in a PR program. Patients who did an adjusted program or a home program instead of the enhanced PR program because of their individual needs, were not included in the study. Sixteen patients were included in the implementation phase and ten patients in the sustainability phase. The characteristics of these participants are depicted in Table 3. In the implementation phase, ve (31%) patients dropped out, because of exacerbation (N = 4) or another illness (N = 1). These patients had a higher CAT-score (mean difference 10.6 points, sd 2.4, p = 0.001) and a higher BODE-index (mean difference 2.7 points, sd 1.1, p = 0.04) which was signi cantly higher than the patients who completed the PR program in that phase. In the sustainability phase only one patient dropped out (Fig. 2).
Not all patients included in the study completed all components of the enhanced program. One patient completed only the education component of the program (implementation phase) and three patients only completed the exercise component (N = 1 from implementation phase, N = 2 sustainability phase). Two patients had the program extended because they had slow progression and would bene t from more exercise sessions.
Adoption, Implementation, Effectiveness and Maintenance: Implementation phase The enhanced PR program was adopted by all HCPs (N = 8, 100%) working in PR program at the MCI, all of whom accepted to participate in the study. Their characteristics are depicted in Table 4. These HCPs participated in the pilot implementation and sustainability study

Abbreviations: SD = standard deviation
The individual education sessions that are part of the enhanced PR program recommendations were not given by the HCPs in the standardized format and could therefore not be assessed for the study. Fourteen group education sessions were scored for implementation delity (Table 5). From these sessions, 76% of the items for content and 95% of the items for delivery received a good delity score (a score ≥ 5). The number of participants present at the education sessions varied between two and ten patients, with a median of four patients. The session 'Integrating an exercise program in your life' could not be scored for the study, because the topic was discussed with the patients during an exercise session when the researcher was not present.
Outcomes related to the program's effectiveness and maintenance at the patient level (the patient outcomes directly post-program and at 3-months follow-up), are depicted in Table 6 and Table 7. Clinically important improvements were shown in functional exercise capacity, knowledge, functional status, self-e cacy outcomes and some self-reported functional performance outcomes directly post-program. At the 3-month follow-up, clinically important improvements were shown in knowledge, self-e cacy outcomes and some self-reported functional performance outcomes compared to baseline. ↓ a decrease of clinical important difference is a positive outcome.
Abbreviations: 6MWT = 6 Minute Walk Test, LINQ = Lung Information Needs Questionnaire, BCKQ = Bristol COPD Knowledge Questionnaire, CAT = COPD Assessment Test, FPI-SF = Functional Performance Inventory-Short Form, MSEES = Multidimensional Self-E cacy for Exercise Scale, SE = Self-E cacy, SEWS = Self-E cacy for Walking Scale, SEAMS = Self-E cacy for Appropriate Medication Use Scale  Self-e cacy MSEES total mean ≥ 10% ↑ 6 (55) 3 (27)  This data was only obtained in the implementation phase of the study. * Number of patients that improved exceeding the clinical important difference. ** Number of patients that did not improve or had an improvement that did not exceed the clinical important difference. ↑an increase of clinical important difference is a positive outcome.
↓ a decrease of clinical important difference is a positive outcome.
Abbreviations: 6MWT = 6 Minute Walk Test, LINQ = Lung Information Needs Questionnaire, BCKQ = Bristol COPD Knowledge Questionnaire, CAT = COPD Assessment Test, FPI-SF = Functional Performance Inventory-Short Form, MSEES = Multidimensional Self-E cacy for Exercise Scale, SE = Self-E cacy, SEWS = Self-E cacy for Walking Scale, SEAMS = Self-E cacy for Appropriate Medication Use Scale Adoption, Implementation and Effectiveness: Sustainability phase The enhanced PR program was still being delivered at the MCI 18 months after implementation. The same HCPs were involved in the program and participated in the study.
During the sustainability phase, 32 group education sessions were scored for implementation delity (Table 5). A good delity score was achieved for 82% of the items for content and 88% of the items for delivery. This was not signi cantly different from the implementation phase. The number of patients present at the group education sessions varied from 3 to 6 patients.
Data on exercise sessions was extracted for nine patients from their medical records. The duration of the program executed by the HCPs from the MCI was 6 or 7 weeks with 15 exercise sessions planned, but varied between individual patients from 5 to 12 weeks, with an average of 7 weeks which is between the recommended duration of 6 to 12 weeks. The number of exercise sessions varied from 9 to 27 sessions with an average of 15 sessions which is less than the minimal of 24 sessions which are recommended for the enhanced PR program. The recommendations of the enhanced PR program regarding home exercise, types of exercise, monitoring and the use of strategies to determine exercise intensity were followed by the HCPs at the MCI.
Patient outcomes from the sustainability phase are shown in Table 8. Functional exercise capacity, knowledge, functional status, self-e cacy outcomes and some self-reported functional performance outcomes showed clinically important improvements for patients directly post-program.
ended questions to get insight in barriers and facilitators for participation in the enhanced PR program as perceived by patients. Barriers that patients mentioned were: 1) bilingual group education sessions, 2) weather conditions and 3) long travel distance. Facilitators were: 1) participating in a group, 2) patient-speci c adjustments of the program, 3) exercise and education sessions scheduled right after each other (reducing travel) and 4) being accompanied by someone.
Overall, HCPs were very satis ed with the enhanced PR program in the implementation phase, with an overall average score of 8.1 out of 10 (SD 1.4) (Additional le 4). They were less satis ed with the introduction of the program (mean score 6.7 SD 2.8) and the facilitator notes and resources for the education sessions (mean score 6.4, SD 3.4).
From the surveys for HCPs from both phases, barriers mentioned for implementation and sustainability of the PR program were: 1) lack of time, 2) having other priorities, 3) changes needed in slides, and 4) lack of French materials. Facilitators were: 1) longstanding procedure for assessments, 2) having a large team with many disciplines and 3) being familiar with LWWCOPD self-management program.

Discussion
This study provides, for the rst time, a full evaluation of the implementation and sustainability of the enhanced PR program. We found that the enhanced PR program was successfully implemented at the MCI and accepted by both patients and HCPs. Adoption was maximal for this study as all HCPs were willing to initiate the enhanced PR program. The program was maintained over 18 months with high program delity of the group education sessions and most of the exercise recommendations. Moreover, the HCPs were committed to further sustainment of the enhanced PR program. Our evaluation also showed that patients had clinical important improvements in functional capacity, knowledge, functional status and self-e cacy outcomes post-program and at 3-month follow-up. These improvements are in line with the body of evidence on the bene ts of PR for individuals with COPD 1,24−27 .
With 70% (implementation phase) to 75% (sustainability phase) of the patients agreeing to do a rehab program, the reach of the overall enhanced PR program was high compared to other studies reporting uptake of PR ranging from 3-70% [28][29][30] . This great acceptability may be explained by the fact that referrers might have screened patients who were more likely to accept the referral to a PR program. There were a number of patients who did not participate in the enhanced PR program due to barriers which prevented them from attending a regular outpatient program. For example patients who were too frail to participate in the outpatient program received a prescription of a home exercise program. This nding highlights the need to improve access to a variety of PR programs that can be delivered in different contexts such as tele-medicine PR and home PR programs 14,31 where the resources of the enhanced can be used as a standard. Measuring the reach of the enhanced PR program on individual (patient) level was limited within this single-centre study and there were no particular strategies used for increasing uptake of the program. In a larger study with multiple centers involved, a comparison of reach between these sites can be made.
Factors that contributed to the successful implementation at the MCI were that the HCPs were already working together for many years in an existing PR program and that they were familiar with the LWWCOPD program prior to implementation. In addition, the MCI has a very high number of professionals dedicated to the PR program (N = 8) compared to data from a Canadian survey in 2015 12 (median 4, interquartile range 3-6). It is expected that the implementation of the enhanced PR program into sites that have fewer HCPs available or with less experience with PR and/or self-management education or into sites where PR is not available yet, will be more challenging. In these settings, new barriers for implementation and sustainability of the program may be encountered. Finally, this project had strong support from the director of the PR program (JB) who was also an investigator in this study. This might have in uenced the willingness of the HCPs to contribute to the project and facilitated implementation. Therefore, it is important to evaluate the implementation of the enhanced PR programs in other settings in the future.
Our study has many strengths. To date, there are no studies that have systematically assessed the implementation of an evidence-based and comprehensive PR program. There is strong evidence for the effectiveness of PR on multiple outcomes in multiple populations 1,8 but none on the process of implementing evidence-based PR into clinical sites 5,11,13 . Implementing evidence-based knowledge into practice is a complex process 32-35 but can improve the quality of health care and patient outcomes 36 . In addition, implementation projects with systematic evaluation are needed to help increase the accessibility of PR for patients with COPD 11 . Another strength of this study is the use of the RE-AIM framework, which ensured a rich program evaluation on multiple levels and on multiple domains in a real-life setting.
One of the limitations of the study is that only one site was included. This resulted in a small sample size and limited the analysis of patient outcomes using statistical tests. However, this was not the main focus of the study and by presenting the MCIDs the study still provided insights on the clinically meaningful effects of the program for patients with COPD.
With the experienced HCPs, the likelihood of a successful implementation was high for this site. Still, this study provides some lessons learned for future implementation of the enhanced PR program, which are: 1) A clearer introduction of the program based on the knowledge and experience of the HCPs on PR was needed, 2) The HCPs wanted to be better informed about the programs' resources (e.g. facilitator notes, presentation slides and reference guides for the group education sessions) and 3) HCPs should be allowed to adapt some of the programs' components for feasibility, such as the order of group education sessions and the slides used for these sessions.
In conclusion, this study showed a successful implementation of the enhanced PR program. Considering the added bene t of the enhanced PR program compared to traditional PR in terms of reducing healthcare utilization 15 , a next step would be further implementation and evaluation of the enhanced PR in a variety of settings including rural, community and tele-health settings which may or may not currently provide PR. The process and framework used in this study to evaluate the implementation of the enhanced PR can be of interest to researchers and clinicians who intend to implement PR programs in their sites. The enhanced PR program can be used by HCPs or healthcare managers to start a new program or increase the quality of an existing program. All resources and recommendations are available without cost on www.livingwellwithcopd.com (sign-up as a professional under the tab Canadian PR program).