The current study utilizes a stepped-wedge randomized design to investigate the effect of the LOCI strategy. Please see the study protocol (19) for further details about the study.
Participants
Participants were therapists (n = 804) with an average age of 43.8, 75.4% were female, and approximately half were clinical psychologists (Table 1). They completed questionnaires addressing implementation climate and general and implementation leadership among first-level leaders (n = 47) who received the LOCI intervention at 43 participating clinics (Table 1).
Table 1
Participant characteristics
|
LOCI leaders
(N = 47)
|
Therapists
(N = 804)
|
Overall
(N = 851)
|
Gender
|
|
|
|
Women
|
29 (61.7%)
|
606 (75.4%)
|
635 (74.6%)
|
Men
|
18 (38.3%)
|
171 (21.3%)
|
189 (22.2%)
|
Education
|
|
|
|
Psychology
|
26 (55.3%)
|
371 (46.1%)
|
397 (46.7%)
|
Medicine
|
5 (10.6%)
|
151 (18.8%)
|
156 (18.3%)
|
Social worker
|
8 (17.0%)
|
60 (7.5%)
|
68 (8.0%)
|
Nurse
|
8 (17.0%)
|
55 (6.8%)
|
63 (7.4%)
|
Other
|
0 (0%)
|
89 (11.1%)
|
89 (10.5%)
|
Age
|
|
|
|
Mean (SD)
|
49.7 (7.64)
|
43.8 (11.1)
|
44.2 (11.0)
|
Missing
|
0 (0%)
|
115 (14.3%)
|
115 (13.5%)
|
Procedures
This study was conducted in mental health clinics for children and adolescents and for adults, which were localized within health trusts across Norway. At baseline, all therapists (n = 804) in the participating clinics were trained in screening and diagnosing of PTSD (Table 2). In addition, a sub-group of therapists (n = 249) received training in three of the most well-documented EBPs for PTSD (34, 35), namely Trauma-Focused Cognitive-Behavioral Therapy (36) for children, and either the Eye Movement Desensitization and Reprocessing (37) or Cognitive Therapy for PTSD (38) for adults. The CT-PTSD and EMDR training consisted of a three-day course followed by 10 hours coaching group calls divided by 2 hours once a month for 5 months. Training and supervision were given by specialists in each of the three EBPs. The TF-CBT training included three days of initial training followed by weekly 1-hour case coaching calls in groups for a year (approximately 40 hours). Following the training, all clinics were eligible to screen patients and provide EBP for PTSD.
Table 2
Stepped-wedge study design
Cohort
|
Time periods
|
Measure 1
|
Training*
|
Measure 2
|
Training*
|
Measure 3
|
Training*
|
Measure 4
|
Training*
|
Measure 5
|
Training*
|
Measure 6
|
Training*
|
Jul
2018
|
Sep
2018
|
Dec
2018
|
Jan 2019
|
Apr
2019
|
May
2019
|
Jul
2019
|
Sep
2019
|
Dec
2019
|
Jan
2020
|
Apr
2020
|
May
2020
|
I
|
Non-LOCI
|
EBP + LOCI start-up
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI graduation
|
Sustainment phase
|
|
Sustainment phase
|
|
II
|
Non-LOCI
|
EBP
|
Non-LOCI
|
LOCI
start-up
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI graduation
|
Sustainment phase
|
|
III
|
Non-LOCI
|
EBP
|
Non-LOCI
|
|
Non-LOCI
|
LOCI
start-up
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI booster
|
LOCI implementation phase
|
LOCI graduation
|
*Training: |
Training in EBP: Training therapists in evidence-based screening and treatment methods for PTSD. |
Training in LOCI: Training leaders in general and implementation leadership, and implementation climate (start-up meeting, booster meetings, and graduation) |
Forty-eight first-level leaders from 48 different child (n = 26) and adult (n = 22) clinics were randomized by a computer algorithm into one of three cohorts, each initiating training in LOCI at three different time points as indicated in the stepped-wedge (Table 2). The stratified randomization was made based on the following variables: number of therapists per clinic, co-localization of more than one clinic, number of therapists to receive training in each of the EBPs, number of therapists per LOCI leader, total number of inhabitants for each randomization unit, number of municipalities or districts for each unit, and number of inhabitants within the health trust served by the participating clinics. Power calculation based on 48 clinics showed that a difference at a little below .4 standard deviations will be detected with 80% power. The random allocation and enrollment and assignment of participants were conducted by the research group.
Four clinics dropped out from the project during initial phase of LOCI (one from cohort 1, two from cohort 2, and one from cohort 3) and were excluded from the main analysis. Linear mixed effects analysis with clustering on clinics demonstrated that there were no significant differences in scores (p ≥ 0.564) between therapists in dropout and participating clinics in terms of baseline scores on implementation leadership, transformational leadership, or implementation climate. The final sample consisted of 47 first-level leaders from 43 different child and adult clinics (due to a change in leadership in three of the clinics, there are more leaders than clinics). Cohort 1 consisted of 14 clinics (16 leaders and 320 therapists), cohort 2 of 14 clinics (14 leaders and 235 therapists), and cohort 3 of 15 clinics (17 leaders and 249 therapists). Please see participant flow in the CONSORT diagram as Supplementary Material.
The LOCI training sessions (2 days at baseline and 1 day at 4, 8, and 12 month) were carried out face-to-face at the Norwegian Center for Violence and Traumatic Stress Studies (NKVTS). During these trainings, first-level leaders were introduced to general and strategic leadership principles and implementation climate. The leaders received feedback reports based on 360° assessments on their leadership and their clinics’ implementation climate. Based on this, they developed individualized leadership development plans to progress toward improvements in leadership and climate which were updated based on new feedback reports every fourth month. The first-level leader had weekly coaching calls by phone with a LOCI trainer to strategize actions to take to achieve the goals defined in the leadership development plan. Once a month, the individual coaching calls were replaced with group coaching calls within each cohort.
The first organizational strategy meeting (OSMs), which involves first-level leaders and executive management, was conducted at each clinic following the first LOCI training sessions, whereas the following OSMs were conducted through digital platforms. Consistent with the LOCI strategy focus on alignment of first-level leader activities and organizational supports, data were shared with executives in each of the health trusts at every OSM. Climate development plans were co-created with health trust executives in order to better support first-level leaders in supporting their therapists in the implementation of EBPs for PTSD.
The LOCI strategy was administrated separately in the child and adult clinics by two teams at the NKVTS. The team for adult clinics consisted of two clinical psychologists and one PhD. The team for child clinics consisted of three clinical psychologists, one MA, and one PhD. Two of the five LOCI trainers in the child clinics were also responsible for training therapists in TF-CBT. Both teams were trained to deliver LOCI by the original LOCI developers. There were regular meetings between the Norwegian and US teams to discuss and review adaptations (such as context and design issues), translation of materials and measures, and fidelity to the LOCI protocol. LOCI’s developer (GAA) participated at the first LOCI workshops and follow-up workshops with both teams, and attended and provided feedback on meetings with health trust executives. In addition, the Norwegian LOCI trainers had regular meetings to discuss the progress during the project period.
We collected data from all participating clinics throughout the study period, consistent with the stepped-wedge design. This was done using the Norwegian Centre for Research Data (NSD WebSurvey). There were 6 total data collection points (baseline in July 2018 and every four months until April 2020). The first two cohorts entered a sustainability phase at measurement times 5 and 6 respectively.
Measures
The employees completed questionnaires about their perception of their leader and implementation climate for their clinic. For all scales, questions were tailored to evidence-based screening and treatment of PTSD, referring to the screening instruments and treatment methods being implemented. The following scales were used (bootstrapping based on 1000 samples applies to all stated confidence intervals for Cronbach’s alpha).
The Implementation Leadership Scale (ILS)
ILS is a 12-item questionnaire measuring leadership for EBP implementation (Aarons, Ehrhart, & Farahnak, 2014). It consists of four subscales: (1) proactive leadership, (2) knowledgeable leadership, (3) supportive leadership, and (4) perseverant leadership. It is scored from 0 (not at all) to 4 (to a very great extent). The total ILS score was created by computing the mean of the four subscales. Individuals who had data on half or more of the items in each subscale were included. The scale demonstrated excellent psychometric properties (12-items; α = 0.955, CI (95%) = 0.945–0.963).
The Multifactor Leadership Questionnaire (MLQ)
MLQ is a 36-item questionnaire which is built on the full-range leadership theory (39). It measures three leadership behaviors, including transformational, transactional, and non-leadership. Of these, the primary focus for this study was transformational leadership. Transformational leadership consists of four subscales (idealized influence, 8 items; inspirational motivation, 4 items; intellectual stimulation, 4 items; and individual consideration, 4 items). The other scales on the MLQ were also included in the analyses for comparison purposes. Transactional leadership (contingent reward, 4 items; active management-by-exception, 4 items; passive management-by-exception, 4 items) and non-leadership (laissez-faire, 4 items) consists of three and one subscales, respectively (40). Items responses range from 0 (not at all) to 4 (frequently, if not always). While the transformational leadership scale is psychometrically supported in the literature (41), the other scales covary differently both theoretically and empirically from standard psychometric representations (42, 43). We therefore created a total score for transformational leadership by calculating the mean scores across the four subscales, while analyzing the other subscales of transactional and non-leadership separately. Participants with data on two or more of the items in each subscale were included.
Psychometric properties for transformational leadership were excellent (20-items; α = 0.958, CI (95%, bootstrapping based on 1000 samples) = 0.948–0.965) while the subscales for Transactional Leadership all had good item reliability, specifically contingent reward (4-items; α = 0.846, CI (95%, bootstrapping based on 1000 samples) = 0.813–0.871), active management-by-exception (4-items; α = 0.881, CI (95%, bootstrapping based on 1000 samples) = 0.859–0.898), passive management-by-exception (4-items; α = 0.842, CI (95%, bootstrapping based on 1000 samples) = 0.812–0.868), and laissez-faire leadership (4-items; α = 0.867, CI (95%) = 0.838–0.892).
The Implementation Climate Scale (ICS)
The ICS is an 18-item questionnaire measuring a climate that supports EBP adoption and use in organizations (26). It includes six subscales: (1) focus on EBP, (2) educational support for EBP, (3) recognition for EBP, (4) rewards for EBP, (5) selection for EBP, and (6) selection for openness. It is scored from 0 (not at all) to 4 (to a very great extent). Participants with data on two or more of the items in each subscale were included, and the total ICS score was then calculated by computing a mean score of all subscales. The ICS showed very good psychometric properties (18-items; α = 0.894, CI (95%) = 0.873–0.910).
The Implementation Climate Measure (ICM)
To include a more global understanding of implementation climate, the ICM, a 6-item questionnaire measuring the general implementation climate in the organization, was also included (44). It includes three subscales measuring what is (1) expected, (2) supported, and (3) rewarded when implementing a new practice. The scale is scored from 0 (not at all) to 4 (often, if not always). As each subscale only contains two items, participants had to have data on all items to be included in the analyses. The ICM total scale score was calculated by the mean scores of all subscales. It showed excellent psychometric properties (6-items; α = 0.918, CI (95%) = 0.901–0.932).
Analyses
All data were exported from NSD WebSurvey to SPSS. The analyses were performed in R (45), using the nmle package (46) for the repeated measures. To assess the internal validity of ILS, MLQ, ICS and ICM, Cronbach’s alpha was calculated using the cronbach.alpha function in the ltm package including a 95% confidence interval using bootstrapping with 1000 samples.
All analyses included data provided by therapists on their perception of general leadership, implementation leadership, and implementation climate. Data provided by leaders were excluded in the current study. In a repeated measures design, responses at the individual-level (i.e., therapists) and responses from individuals within the same clinic are likely to be correlated. To account for the dependency in the data, we used linear mixed effects models which allows for irregularly spaced measurement time periods (47), and missing data within measurements (48), with fixed effects representing different linear changes before and during the LOCI intervention, and random effects for differences between clinics, and differences in level and slope between therapists. The random structure was simplified when necessary for model stability, as recommended (49). The gap between the two linear fits, evaluated when LOCI began, represents the initial impact of LOCI training, where a positive value indicates improvement. Standardized versions of the initial impact, termed d, are computed by dividing by the square root of the combined variances for random effects in levels. If the post-LOCI slope is higher than the pre-LOCI slope, it indicates that the effects of LOCI training increase over time.
In order to examine possible differences between cohorts, training (i.e., received training in screening tools only or both screening tools and the EBPs for PTSD), and outpatient clinics (i.e., child or adult psychiatric care), we added categorical variables for cohorts, training, and outpatient clinics to the model in supplementary analyses. Separate parameters were included to test whether the LOCI training and the pre- and post- trajectories were different across child and adult clinics. Details on these supplementary analyses were only given if there was a significant interaction.