In this study, a mixed methods approach was used combining a qualitative concept mapping workshop with cross-sectional survey data to advance the understanding of organisational implementation climate in implementing iCBT services in mental health care settings. This combination of empirical methods was used to obtain a qualitative understanding of how implementers characterise organisational implementation climate and substantiate this with a quantitative exploration amongst mental health service deliverers in an organisational context in which real implementation took place.
The top-3 ranked characteristics of a strong organisational implementation climate conducive of improving implementation outcomes, include: (1) clarity on role and skills of implementers, (2) feasibility of implementation targets, and (3) instigating a dedicated implementation team. Furthermore, various practical tools can be used to build a strong organisational implementation climate through (1) regular and structured job performance feedback, (2) structurally monitoring progress, and (3) guidelines and methods for impact assessment. These main findings from the concept mapping workshop are aligned with Klein and Sorra’s integrative model of determinants of the effectiveness of organisational implementation [24]. In their model, implementation effectiveness is in part a function of the strength of an organisation’s climate for implementation which comprises a set of organisational policies and practices. According to this theory, different organisational policies and practices may be equifinal in their outcome, skills and motivation play an important role in achieving sustained use of the innovation as unskilled, unmotivated are unlikely to use the innovation at all [24]. This aligns with what implementers ranked high in the workshop regarding roles, capabilities and skills of implementers, implementation targets, and the competences of the implementation team as a whole. Similarly, these findings are aligned with the Normalisation Process Theory (NPT) which takes a sociological perspective in theorizing the way people act and interact in integrating and embedding new ways of working in existing practices [55]. For example, the importance of skill sets in organising collective action, corresponds with the finding that for implementers to be effective, they need to have the position and role in the implementation work and team that fits their capabilities and skills. In addition, having realistic implementation time frames, and practical and feasible targets can influence how the new intervention is used in practice. This corresponds to NPT’s notion of interactional workability as a factor shaping collective action through operationalization of the innovation into practical ways of working that fit the local context. Furthermore, the finding that members of an implementation team should have a shared interest and beliefs in the implementation goals, corresponds to the theory’s notion of coherence referring to processes of individually and collectively determining the innovation’s practical meaning and utility.
Turning to the cross-sectional survey, mental health service delivers generally are satisfied with iCBT (MCSQ = 9.11, SD = 1.96) and regard usability of the iCBT services as slightly below average (MSUS = 63.76, SD = 15.53). Furthermore, the participating mental health service deliverers regarded the organisational implementation climate they operate in as supportive to implementing the iCBT services (MICS = 43.21, SD = 5.62). These acceptability scores are slightly more positive than existing literature on clinicians’ perspectives toward delivering Internet-based psychotherapies. In a German survey comparing acceptance of web-based psychotherapy, it was found that clinicians scored around the summed midpoint of the scale (total score = 45.18, scale range = 16–80, n = 428) indicating a more neutral stance [56]. Another study found an overall a neutral stand point (M = 3.45, SD 0.72, 5-point Likert scale with 3 as neutral score, n = 95) on a survey designed to contrast perceived advantages and disadvantages of Internet-based therapies among Austrian psychotherapists [57]. A third study reported similar score patterns of perceptions of computer-based psychological treatments (M = -0.05, SD = 0.79, 5-point Likert scale with 0 as neutral score, n = 26) [58]. This difference in perceived acceptance of Internet-based psychotherapies might be explained by that the majority of the service delivers (82%) involved in the MasterMind project received iCBT training prior to filling out the survey whereas 80% of the participants in the Schröder study indicated to have no or limited prior knowledge of Internet-based interventions. This might indicate the samples were drawn from different groups of mental health service providers and the possible difference between intended use by non-experienced professionals and actual use by trained professionals. In addition, the difference in findings might be due varying study designs applied. In our study we choose to use more generic instruments (SUS and CSQ) whereas in the other studies applied questionnaires that were specifically developed for the studies’ purposes.
A third finding in this study is that a stronger organisational implementation climate is (weak to moderate) associated with higher levels of satisfaction and usability of iCBT. Although causality is not proven by this study, this finding could lead to proposing that acceptability of iCBT services in terms of usability and satisfaction might vary as a function of organisational implementation climate. That is, stronger organisational implementation climates might support higher acceptance of iCBT services by mental health service providers. This is aligned with an earlier finding of Aarons and colleagues [25] concluding that organisational climate is associated with mental health service providers’ attitudes towards deciding to adopt evidence-based practices in general. This American study amongst public sector professionals providing youth and family mental health services, showed that strong organisational cultures for implementing evidence-based practices was associated with positive attitudes of participants towards those practices. Similarly, a weak organisational implementation climate was associated with higher levels of perceived discrepancies between current and new ways of working, most notably when there are unclarities and conflicts about roles and responsibilities. Authors conclude that clear specification of deliverers’ roles and actions can enhance implementation climates and subsequent contribute to implementation success. This finding also aligns with findings from our conceptualisation workshop, notably as ranked number 1 characteristic of a strong organisational implementation climate conducive of improving implementation outcomes.
When viewed in combination, the qualitative findings from the workshop on the characteristics of a strong organisational implementation climate conceptually align with and validate the survey measurement approach. Despite the pragmatic nature of the survey, items of commitment, attitudes, and resources measured in service delivery staff conceptually align to implementers’ notions of people and skills, the implementation team, availability of resources and attitudes. For example, attitudes as referring to the perceived self-esteem in using a new intervention found in the workshop, directly corresponds to a survey item about confidence in ones’ own ability to implement. Similarly, the importance of resources supportive to the implementation work such as incentives, skilled people and champions, time, technology, technological support, and policies, qualitatively aligns to survey items addressing availability of qualified staff, adequate resources, and implementation strategies. In that respect, the findings of the workshop seen in light of the survey suggests that organisational implementation climate is not only an inherent property of the context in which implementation activities take place, it can also be intentionally shaped to enhance impact of those activities.
Strengths and limitations
Combining a qualitative conceptualisation workshop with implementers with a quantitative cross-sectional survey amongst mental health service delivers can be regarded as a strength of this study as it enables illustrating different aspects of organisational implementation from different viewpoints. This study contributed to a refined understanding of organisational implementation climate in mental health care settings from the viewpoint of implementers, as well as a quantified perception of organisational implementation climate amongst those who – are required to – actually deliver innovative iCBT services. By combing these different viewpoints in one study, a more complete picture of organisational implementation climate in relation to implementing iCBT services in mental health settings is provided.
However, the findings of this study should be interpreted with care for several reasons, including the inevitable heterogeneity in the settings in which the organisations implemented these iCBT services, and the representativeness of implementers and service deliverers in implementing and delivering the services. Regarding the implementation settings, service organisations not only varied in their position in the regional health care system (primary, secondary care), they also varied in their sources of funding for delivering mental health services (see Table 1) driven by underlying regional and national policy contexts. Although in general, most mental health service organisations in Europe are transitioning towards deinstitutionalised care [59], the organizations participated in the MasterMind project likely had differing objectives in implementing the (self-selected) iCBT service. In relation to that, it must be noted that solely taking part in the MasterMind project and receive (complementary) European funding for implementing and evaluating iCBT services, might have impacted decision making on implementation activities and their respective outcomes. Furthermore, the implementers at the organisations recruited the respondents for the survey which might have led to a biased sample of service deliverers who had an interest in innovation and international collaborations in the field of mental health.
In addition, other methodological strengths and limitations of the concept mapping workshop and survey need to be considered. The workshop by which the brainstorming and ranking was achieved, was highly structured. Participants received instructions in advance of the meeting, a combination of pen-and-paper and digital recording methods were used, as well as individual silent idea generation and rankings and structured one-by-one group clarification discussions were used to prevent production blocking [60]. The workshop was held in English. Because only two participants were native English speakers, cognitive inertia might have been induced pursuing participants to the same line of thinking and potentially leading to fear of being judged and pressured to remain within the scope of existing options. Although the workshop was designed to include silent individual and group work, this pressure might have influenced the performance of the group in generating a rich variety of ideas during the first two steps and ranking of ideas later on. A combination of offline and online methods involving both experienced implementers, researchers and service deliverers in for example a Delphi method could enrich the findings presented in this study. For the survey and as noted before, causality between acceptability and organisational implementation climate cannot be inferred from the current study. We merely explored if providers’ acceptability of iCBT services in terms of usability and satisfaction, empirically varied with perceived organisational implementation climate. Further experimental research is required to test if organisational implementation climate moderates perceived service satisfaction and usability and what practical tools are most effective in increasing acceptance and uptake of iCBT services. In addition, reliability, accuracy and applicability of the ICS scale might be limited due to the pragmatic nature of item development and selection process.
Future research
In this study, a notion of the value and nature of organisational implementation climate in implementing iCBT services in routine mental health care has been explored. To further this line of research, open phenomenological research is required focussing on further theorising the mechanisms by which organisational implementation climates exerts change in implementing Internet-based psychotherapy in mental health settings. Subsequently, controlled implementation interventionist research can empirically confirm the theoretical assumptions and improve our understanding of the complex interactions between the iCBT, implementers, service deliverers and the organisational context they operate in. In this respect, one important research question could be concerned with how and to which extend organisational implementation climates can be used as an active implementation strategy for it to effectively improve implementation outcomes.