Importance of internal consistency in implementation programmes. A qualitative study of focus-group interviews.

Background Programmes for implementing evidence-based treatments (EBTs) reach their goals if they succeed in institutionalizing the EBTs into routine clinical practices. Resistance to change is a common response to such programmes, undermining their ultimate output. This also happened with Ostrobothnia Depression Programme (ODP) in Finland, which was targeted at several regional psychiatric teams. We conducted a process evaluation of it to ascertain how managerial and executive practices enable or inhibit implementation and how to overcome resistance.Methods We conducted focus group interviews to scrutinize the phases of preparation and practical execution of the ODP from the perspectives of management and programme executives. To gather the data, we applied the revised Socratic approach for health technology assessment and the rapid assessment process model. We analysed the data deductively according to the Normalization Process Theory.Results We identified two main critical issues: 1) The ODP comprised two simultaneously administered programmes, one for the implementation of two EBTs and the other to study their effectiveness. Tension between these two programmes was apparent right from the beginning. 2) The programme had a simplistic top-down design and, in the preparation phase, collaboration with its target groups was delayed and minimal. No formal implementation theory or framework was applied. In addition, independent internal processes in the separate teams influenced their reception of the ODP.Conclusion We recommend early collaboration between team leaders and prospective treatment providers right from the preparation phase of an EBT implementation programme. Agreement on goals and the instruments for achieving them is essential. The strategies for execution should be adjusted with respect to the prevailing implementation climates within the separate target teams. A collaborative approach may enhance the implementation climate and mitigate resistance. Applying some appropriate implementation theories and frameworks may facilitate that work.

to increase readiness for change among the prospective participants We found the Revised Socratic Approach for Health Technology Assessment a feasible instrument for the process evaluation of an implementation programme.
Quality improvement is the main aim of a programme for implementing an evidence-based treatment (EBT) in the context of a health care organization (1). The ultimate intended beneficiaries are the patients. The key challenge for management and programme executives is to develop a programme plan prompting frontline treatment providers to adopt the EBT into their routine practices (2). Several theory-based implementation models or frameworks have been constructed to facilitate the work (3)(4)(5)(6)(7)(8)(9). The COM-B framework states that the programme should enable providers to achieve better capability (C) or opportunity (O) for performing their work. This serves to enhance their motivation (M) to apply a certain behaviour (B) (6).
Several determinants controlling the acceptance of an implementation programme have been identified (5,10), these include whether the programme design is top-down or bottom-up, whether collaboration with each stakeholder group is early or late, and the reactions to various manifestations of the readiness for change among the relevant personnel (11)(12)(13). The role and performance of leadership have been determined to be critical for the success of the programme and also for sustaining its outcomes (2,14,15). 'Programme theory' is a concept that refers to an individual idea about what might be achieved by which interventions or operations in a given context (16). This theory, in turn, guides those responsible for the programme in designing the programme plan. They may accomplish this work either heuristically, relying on their previous experience or expertise and methodically, grounding their design in a theory-based framework or model (7).
The administrative management of the psychiatric department of South Ostrobothnia Hospital District in Finland launched the Ostrobothnia Depression Programme (ODP) (17). The main goal was to bring about a change in the clinical practices to bridge a gap between the resources available and the increasing demand in the treatment of depressive patients. The ODP was a regional programme comprising two concurrent programmes: the Ostrobothnia Depression Study (ODS) and the related Implementation Programme (ODS-I). The ODS-I was aimed at frontline therapists implementing motivational interviewing (MI) and behavioural activation (BA), both evidence-based brief psychotherapy interventions (18,19). A quantitative evaluation of the ODS-I showed that a third of the target group of the programme were active adopters of MI and BA (20). The ODS was intended to study the effectiveness of MI and BA in a naturalistic setting and the results were positive (21).
As a part of the quantitative evaluation, we realised that the ODP lacked strategies for sustaining implementation outcomes in the long term (20). This was due in part to the weak role of the team leaders in the execution of the ODP, which was highlighted in our mixed-methods process evaluation among the frontline therapists (Lindholm et al., submitted). In addition, substantial resistance to change was encountered in some of the participating teams. These observations led us to augment the process evaluation qualitatively with special reference to the perspective of management and the programme executives. We formulated our research questions collaboratively with the clinical director and the principal programme executive: How do managerial and programme executive practices enable or inhibit the implementation of health innovations or reforms in the clinical context?
How to overcome resistance to change?

Theoretical instruments
To create the interview protocol, we applied the Revised Socratic Approach for Health Technology Assessment (HTA) (22). When designing the iterative data collection process, we applied the Rapid Assessment Process model (RAP) (23). We modified the methods to meet the needs of the present study. To guide the deductive qualitative content analysis (24), we applied Normalization Process Theory (NPT) (25,26). In reporting the study, we adhered to the 32-item checklist of consolidated criteria for reporting qualitative studies (COREQ), which is presented in Additional file 1 (27). The Revised Socratic Approach for HTA, the RAP and NPT are introduced in Additional file 2.

Setting
The ODP ran during the period 2009-2013. The hospital district implementing the ODP is responsible for the provision of public specialized health care services to a population of 200,000. The ODP was carried through in six psychiatric units, of which five were under the administration of the managing organization and one external. The external unit was excluded from the present study in order to focus the evaluation on intra-organizational processes and to avoid confusing the effects of interorganizational interactions. We collected the present data in March 2015, 16 months after the end of the ODP. The time gap was because the analyses of the final summative inquiry and mixed-methods process evaluation, both administered to the frontline therapists in spring 2014, revealed a need to complement our understanding about the realization of the ODP (20,28). The present authors' relations to ODP and the managing organization as well as their mutual professional relationships are presented in Additional file 3.

Forming the study group
To assemble the study group according to the purposeful sampling strategy of the 'complete target population' (29), we emailed the whole ODP project group and all team leaders of the target units, 14 individuals in total. Only one recipient involved in the project group declined the invitation due to compelling personal reasons, thus resulting in a study group of 13 individuals. We informed the study group in advance about the purpose, setting and course of the study as well as the principles for handling the data. This included information about the videotaping of the interviews and the assurance that no interviews would be transcribed due to the sensitive nature of the material and further the assurance that each participant's identity would be protected as far as possible during processing and utilization of the information obtained. Recipients were assured that participation in the study was voluntary and would in no way affect their status within the organization. All members of the study group gave verbal consent to participate.

Description of the study group
The study group was divided into two Focus Groups (FG1 and FG2) according to each member's relation to the ODP. The FG1 comprised the ODP project group, altogether five people, namely the clinical director of the psychiatric department, the principal designer and executive of the ODP, a professor of psychiatry, an associate designer and an executive who was a registered psychologist and two assisting research nurses. The FG2 comprised the team leaders, both psychiatrists and registered nurses, altogether eight people. All members of the study group and the researchers had been regularly employed in the organization for years before the launching of the ODP, thus their relationship was established prior to the present evaluation.

The semi-structured focus group interviews
We interviewed the FG1 twice (FGI1.1 and FGI1.2) and FG2 once (FGI2) (Figure 1). Each interview lasted three hours and was divided into two parts with a short break between them. Four members of the FG1 and five of the FG2 attended the group interviews in person. Four individuals were unable to attend the group interviews in person due to pressure of work, so they provided the desired information in alternative ways: The FG1-enrolled associate executive was interviewed separately immediately after the FGI1.1 and the information obtained was included in the relevant report. One FG2-enrolled person provided written feedback before the FGI2, and this information was presented to the FG2 during the interview. The remaining two FG2-enrolled people who were unable to attend in person had discussed the issues beforehand with their attending colleague.
The first author acted as a facilitator in the interviews with FG1 and second author took notes. The FGI2 involved only the first author also taking notes while acting as the facilitator, since second author had to be excluded due to her managerial relation to the nurse members of the FG2.
Videotaping made it possible to check the notes afterwards.

The interview guide
The same interview guide was used for the interviews with FGI1.1 and FGI2. It comprised five themes and nine guiding questions relevant to the ODP. We constructed the themes to encompass information of interest; they concerned underlying motives and intentions, management, the perspective of the participating units, the interests of individuals conducting the present evaluation and creating the vision for future developments. We constructed the guide by selecting relevant explanatory questions on the themes above from the Revised Socratic Approach for HTA (22). Next the original explanatory questions selected were reformulated for the purposes of the present evaluation and renamed as guiding questions. (See Additional file 4, Table A).
The interview guide for the FGI1.2 was composed so as to involve the FG1 in the process analysis of the ODP reflexively rather than as being a source of data collection only (see Additional file 4, Table   B).
While creating the interview plan and guides, the first author had reflective discussions about the mission with the second author, the clinical director and the principal programme executive according to the idea of RAP (23). Due to the setting, we had no opportunity to pilot the interview guides in practice.

Creating the raw data
As a base, we had the technical data on the ODP comprising the implementation plan of ODS-I (20), the research plan of the ODS including the protocol for data collection and total executive resources in ODP (see Figure 2 for resources). We gathered the supplementary information through an iterative and collaborative process of FGIs and finally wrote one, rich narrative (in Finnish) on the realization of the ODP that comprised the raw data (in more detail see Figure 1). We organized the raw data according to the following scheme drawing on the framework proposed by Rise et al: organizational structures, management, decision-making processes, cultural change, permeation throughout the organization, anchoring responsibility, prioritizing of resources, understanding of staff involvement and hindsight (30).

Qualitative content analysis
The case of our study was the process of running the ODP all the way from its rationales to its completion, and the unit of analysis was the narrative formed by the raw data (29). We analysed the raw data through deductive qualitative content analysis (24) guided by NPT (see Additional file 5 for coding frame) (25,26). Our analysis and extracting the results progressed in four steps: First, we encoded the raw data using different colours and reorganized it according to the main categories.
Second, we re-encoded and organized the data further according to the subcategories. We reviewed the relevancy of the encoding during the two first steps and readjusted when needed. Third, we condensed and rewrote the information contained in the encoded text pieces into a fluent narrative in terms of each subcategory. Fourth, we answered the research questions on the basis of the data analysed, thereby providing the results of the present study.
The first author performed the process of analysis and extracted the results in close consultation with the second author. Finally, we presented the results to the FG1 for appraisal and possible amendments. They suggested some refinements and, after these have been made, they accepted the results presented below. The analysis of the data was processed manually with assistance of Word for Mac 2011.

Results
Two main critical issues arose from the data, which we interpreted to shed light on the friction in the progress of the ODP and to substantiate the answers to our research questions. The issues were: 1) There was tension between simultaneously administered implementation and clinical research programmes right from the beginning. Although the programme executives explicitly articulated that implementation and quality improvement were the primary intention, the ambitions related to the clinical research practically outstripped those of the implementation programme. 2) The programme theory was grounded on the conception that the goals of the ODP were feasible by addressing programme strategies almost solely to frontline therapists ( Figure 2). The idea of learning by doing supported this approach. The programme theory was purely heuristic and implicit and was not tested

Coherence -communal specification: quality improvement of work or clinical research?
The reasoning and determination of the goals of the ODP were constructed mostly at a high level in the organization (Table 1). Prior to the ODP, the clinical director was aware of increasing distress among the frontline personnel due to accelerating patient flow and the ODP was launched to tackle the problem. The TTG comprised the clinical director, the principal and associate programme executives and a senior consultant but no lower level leaders or frontline therapists from the intended target group. The team leaders were not involved until the phase of finalizing the programme plan.
The primary goal with which the group tasked in the ODP was to achieve quality improvement in clinical practices. In the experience of the team leaders, however, the preparatory process of the ODP had proceeded one-way, top-down, which they considered was a deviation from the normal collaborative two-way managerial practices adhered to while preparing organizational strategies.
They saw one-way preparation as a normal and acceptable practice for research programmes. In addition, in the name of the programme, the term "research" preceded the term "development", which, they said, led their conception in that direction, too.
[insert Table 1  the patient recruitment needed for the clinical research. The largest unit was invited according to the second criterion only, that is, to satisfy the needs of the research. Since they initially declined, they were persuaded to participate after a one-year delay. The other units accepted the invitation at the first step.

Cognitive participation -enrolment: adherence and resistance.
Most of the voluntarily participating units' team leaders saw the ODP as an opportunity to learn something new and to review the prevailing treatment practices, although they saw it primarily as a research programme. The reception of the ODP between units varied from welcoming it, through confusion up to considerable resistance. The programme executives identified one team where the collaboration had been smoothest. The ideas in the ODP were congruent with the team's own ideas, which they had already been working with. By contrast considerable resistance arose in the largest unit, which had to be persuaded to participate. Tackling the resistance greatly depleted the executives' resources. The team leader of this unit deemed the goals for the ODP to be relevant but considered that the change aimed at was too ambitious to be loaded on one programme. Moreover, the team leader appraised merging programmes for implementation and clinical research as an improper setting to reach the goals.
Despite enrolment in the ODP at the level of units, the enrolment of the therapists in the training varied widely between the units. At best, all the therapists of one unit completed the training. At worst, only one or two therapists of a unit joined in, including one temporary substitute. Some of the units assigned more therapists to the training later on and some of them were motivated to speed up the termination of the ODP.

Collective action -skill set workability: division of labour.
Due to at least two reasons the number of patients to be recruited for the clinical research was accumulated more slowly than anticipated:1) some of the initially keen therapists got tired in the course of the process and withdrew and 2) staff turnover cut down the number of ODP-trained therapists. Recruiting patients began to accumulate on fewer shoulders, which caused stress. The question, "when will this be over?" arose among the therapists.

Reflexive monitoring: what happened and why?
Some positive experiences in the early phase caused the programme executives to think that the strategies applied in the ODP had the potential to bring about the desired cultural change in treatment practices at the level of the entire department. However, they became unsure as the programme proceeded, partly because they noticed that patient recruitment for the ODS occupied too large a role and the idea of implementation faded. The team leaders shared the perception that the concurrent implementation and clinical research programmes caused confusion among the therapists.
The number of patients pursued for the clinical research was intended, besides conducting an effectiveness study, to ensure a sufficient amount of practice needed to acquire skills in BA and MI.
The drive to satisfy the scientific interest escalated as the ODP proceeded and this exacerbated the therapists' sense of being under pressure, which further increased their negative perception of the ODP.

Discussion
Our present results highlighted two important dimensions on which the success of a programme depends: 1) Coherence between what has been explicitly communicated and what has been practically accomplished on a programme. The ODP was communicated primarily as an implementation programme for EBTs. However, the target teams perceived the scientific focus to be paramount. 2) The programme theory was too narrow, focusing the programme strategies only on the intended treatment providers. This corroborates our previously published findings that the ODP lacked focused strategies for coaching the team leaders in managerial practices to promote the programme (20). The programme theory of the ODP ignored the team leaders' central role in influencing the implementation climate and mobilizing organizational strategies (2,15).
The present results established that the reception of ODP varied from positive to negative and with confusion in between. According to the positive involvement, the ODP fostered the spontaneous developmental efforts made by one team prior to the ODP. This made the implementation climate receptive to the ODP. In this particular team, the programme was deemed to provide them with an opportunity to improve their professional capability, which, in turn, fuelled their motivation to change their professional performance in the desired direction. This cascade is in accordance with the programme authorities' originally anticipated impacts among the therapists (Figure 3) as well as the COM-B framework (6). In this case, the two programmes, ODS and ODS-I, were mutually supportive.
The negative experience arose from the notion that the ODP had bitten off more than it could chew, which led to an unreceptive climate. This, in turn, gave rise to a perception of the two programmes being in competition with each other, which culminated in a feeling of administrative pressure. This caused frustration and rejection among the staff, which can be seen as negative manifestations of cognitive participation and collective action according to NPT (25). Addressing the therapists' negative experiences required extra attention on the part of the programme executives.
Tackling resistance is, at least partly, a question of collaboration (2,14,31). A collaborative approach that engages all levels of stakeholders at an early phase of defining the problem to tackle and designing the means to achieve the jointly set goals has been shown to be more fruitful than a simplistic top-down designed development programme (11,12,31). Early collaboration intended to create a shared vision of what and how a programme will be aimed to achieve is more likely to meet with a more positive reception among the intended addressees. This would achieve better coherence between stakeholders and an improved drive towards cognitive participation and collective action (25). We recognized that the ODP was top-down in nature with late collaboration, which carries a higher risk of talking at cross purposes and, hence, of resistance among the programme addressees (11). Underestimating the impacts of variation in readiness for change among the target group is one possible mechanism leading to late collaboration (5,13,32). Early collaboration may initially have caused the start-up of the ODP to be more complex and time consuming and, consequently, to require investing more resources. Indeed, those investments may have been recouped later on in terms of a smoother-running programme.
Evaluating programme outcomes is important to ensure that the direction is right (1,10). Possible relevant windows for the evaluation in the context of health care are many: the views of service providers, the implementation achievements and adverse effects, and the patient experiences and treatment outcomes (1). While assessing the quality of health services, all these aspects are important. Studying the effectiveness of an EBT is related to its justification. However, confusion may ensue if individual stakeholders understand various objectives to be loaded differently on a programme. For instance, the impression that different objectives are actually competing against each other may arise (10). This happened inadvertently in several units participating in the ODP in terms of the simultaneously running ODS-I and ODS. The original idea of the ODP was that these two would be mutually supportive. Adjusting the ODP as a whole with respect to the implementation climate in the various units would also have entailed adjustments in administering the effectiveness study.

Strengths and limitations
We reached all but one out of the intended informants since we accepted other ways of providing information than only individual attendance at the FGIs, which ensured obtaining a wide range of opinions. On the other hand, one more iteration with both FGs and inviting a third focus group from the frontline therapists would have provided us with richer data. Also, in not transcribing the interviews we deviated from the conduct of the conventional qualitative interview study. We made this decision as we were interested in the data verbally articulated, not the nonverbal data. These restrictions enabled us to keep the research process within our resources. In spite of these limitations, we consider that the present results contribute to the knowledge of the risks inherent in conducting an effectiveness study in connection with an implementation programme. In addition, we extend the existing knowledge about the need to ensure the collaboration of every stakeholder group.

Fidelity of the data
Special attention was paid to the general climate during the FGIs and to ensuring that the data articulated on the questions of interest was clearly expressed (Massey, 2011). During the interviews, a free and frank dialogue was achieved, where both disagreement and consensus within and between the groups were accepted. A report on each FGI was written only a few days after the interview and sent for confirmation to each participant in the FGI concerned. All reports were approved as such. In addition, the facilitator checked the reports by watching the videotaped interviews and no new substantive information was detected although some amplificatory and descriptive details, e.g. quotations, were indeed picked up. The foregoing serves to verify the true correspondence between the essential contents of the FGIs and the raw data. In addition, the members of the FG1 reviewed the present results section, which was amended according to the feedback.

Conclusion
Our present qualitative process evaluation of an EBT implementation programme highlights that the quest for a shared understanding and agreement on the goals between different stakeholders and the instruments for reaching this is of paramount important. Applying the appropriate implementation theories and frameworks or models may facilitate that work. The evaluation of the programme from different aspects should be adjusted to ensure that the primary goal, the implementation of the EBT, is indeed achieved. We recommend early collaboration with the team leaders and intended treatment providers in order to elaborate a shared programme theory and to design a programme plan which matches the prevailing implementation climate. This will likely entail more effort in the planning phase but should increase the probability of creating an improved programme more fluently executed and in eliciting motivation among the addressees to change their professional practices in the desired direction.

Ethics approval
According to the principles agreed on in the South Ostrobothnia Hospital District, this process evaluation of the ODP was exempt from ethical approval as the work was primarily intended to improve regional care through the implementation of EBTs and by using personnel members as informants.

Consent to participate and for publication
All members of the study group gave their verbal consent to participate in the study, which included consent to the publication of the analysed data.

Availability of data and materials
The original datasets generated and analysed during the present study are not publicly available due to the requirement to preserve confidentiality. However, the final narrative about the OPD, that is the raw data, is available from the corresponding author on reasonable request. The raw data is in Finnish. We made the English translation during the fourth step of the analysis or answering the research questions.

Competing interests
All authors were employed in the organization where the ODP was implemented. Also, they had various professional relations with each other. These are disclosed in more detailed in Additional file 3.

Funding
The South Ostrobothnia Hospital District research fund supported the ODP but did not influence the course of the research at any stage or preparation of the manuscript.

Author's contributions
LHL designed the present study in close collaboration with OK and AL. In addition LHL led the data collection, constructed the raw data and performed the qualitative analysis as well as made the preliminary interpretation of the data. LHL also wrote the manuscript. ML was closely involved in the data collection, the raw data construction and its analysis and interpretation. In addition, ML reviewed several versions of the manuscript. OK and AL were also informants in the study and according to the study protocol they contributed to the interpretation of the data. In addition, OK reviewed several versions of the manuscript. More related information is presented in Additional file 3: The authors' professional relations to ODP, the organisation and each other.