Study context and trial design
This study uses longitudinal data from nursing staff working within seven inner city, acute in-patient wards, and one specialist in-patient women’s service, in a mental health National Health Service Foundation trust. These data were collected for the DOORWAYS trial (8), which was funded and ran from February 2008 until April 2010. DOORWAYS was stepped wedge, cluster randomized controlled trial (RCT) which was designed so that at each randomisation point, two wards were assigned to the therapeutic intervention arm and those on the 'waiting list' provided a control (see figure 1). Data were collected pre-randomisation, so that randomised wards also acted as controls. This meant that at the T1 data collection point, for example, staff on wards 4 and 8 did not know that they would be receiving the intervention next. Those on wards 3 and 5, were aware that they were in the intervention group, and all others were not aware of when they would receive the training. Altogether, there were five randomisation points, which allowed all of the wards to receive the intervention during the course of the trial. The order of ward allocation to the intervention was determined by a randomly generated list, which was computed by a statistician using the ralloc procedure in Stata, which is a statistical software package.
This study is concerned with only two time periods; baseline (T0), when no wards had received intervention, and the 12-month follow-up, when 4 out of the eight wards had received the intervention, and were running groups.
After randomisation, a clinical psychologist offered training to nursing staff in a number of groups and evidence base activities which included:
Communication skills – to improve communication between staff and service users, and communication more generally.
Social Cognition & Interaction Training – to improve service users' understanding of social situations and minimise misunderstandings with others (24).
Wards were also offered a choice of therapeutic activities, which they selected based on their service requirements, as follows:
Hearing Voices Group - to reduce the distress associated with hearing voices, and to teach new coping skills whilst improving self-esteem (25).
Self Esteem and Coping with Stigma - to reduce the stigma associated with mental health problems, including the negative self-evaluations which may maintain low self-esteem (26).
Emotional Coping Skills - to teach skills to service users for coping with overwhelming negative emotions (common in those who self-harm) (27). This group was based on dialectical behavioural therapy.
Relaxation Techniques – to teach progressive muscle relaxation techniques and breathing exercises to service users in preparation for sleep (26)
Problem Solving Skills – to teach structured methods for problem-solving and involved identifying the problem, brain storming possible solutions, and selecting the best solution(s) (28).
Implementation followed a change management strategy adapted from 'Diffusion of lnnovations' (29). The aim was to identify enthusiastic individuals as champions, which would motivate other members of the team to adopt the intervention. After six months the groups were expected to run regularly because a majority of staff had been trained and involved. After the training, there followed a process of establishing the groups on the wards, through demonstrations by the psychologist. The nursing staff were then asked to deliver the groups independently by the third month. Until the end of six months the psychologist was available for advice and ongoing support. By the twelfth month, all four wards included in this study had received training in communication skills and the intervention groups were running as outlined in figure 2.
All permanently employed nursing staff on acute in-patient wards were eligible to take part in this stage of the study, including staff from band seven (team leaders), band six (clinical charge nurses), band five (entry level qualified staff) and band three (health care assistants).
To estimate the number of participants necessary for multi-level regression models we followed the general rule suggested by Green (30) of ten cases per variable. Given N=120 participants were included in a regression model with five variables, this sample was sufficient.
Ethics, consent and permissions
A local NHS Research Ethics Committee (07/H0809/49) awarded ethical approval for this study. Participants were provided with information sheets and given time to consider participation before providing written, informed consent.
Staff were recruited to each time point over 30 days by an on-site team of research assistants. All staff measures were completed by self-report. Although it was possible for the same staff to participate at multiple time points, changing shift patterns meant that those who participated at baseline were not necessarily available at follow up, leaving the dataset susceptible to losses. The baseline data were collected in March and April 2009 and the follow up data were collected 12 months later.
Primary outcome measure: Staff Perceptions of Barriers to Change:
The 18 item Views Of Change and Limitations in In-patient Settings (VOCALISE) measure (22) was developed with direct participation by mental health nurses to capture the reality of working in wards, and multiple causes of resistance to change. Some “barriers to change” reflected organisational difficulties:
When it comes to change, information is not circulated effectively on my ward;
I’m too busy to keep up to date with information about the changes that are happening on my ward;
Poor leadership prevents changes happening on my ward
Inadequate staffing prevents changes being successful on my ward.
Some described staff reluctance and withdrawal:
VOCALISE is scored on a 6 point Likert scale ranging from strongly agree to strongly disagree. The highest score is 108, and the lowest is 18. In this study, VOCALISE was reverse scored so that high scores indicated negative perceptions. It can be accessed at www.perceive.iop.kcl.ac.uk.
Secondary outcome measures:
Occupational status: categorized into two groups 1) direct care staff and 2) managers. Direct care staff were healthcare assistants and band 5 qualified nursing staff. Managers were bands six and seven nursing staff (i.e. clinical charge nurses, practice development nurses and team leaders).
Ward climate: an eight-category “ward” variable was used as a proxy measure for ward climate.
Time: two time points were included (baseline, 12-month follow-up).
Treatment group: two groups participated: (intervention and control).
As there were a large number of missing data at follow up (only 43% of the baseline sample were repeat participants), we adopted unstructured multivariate linear models.
Unstructured multivariate linear models use both baseline and follow-up data as the correlated outcome, enforcing a zero treatment effect at baseline, with an unstructured covariance matrix for baseline and follow-up measures (31-35). The models allow more information from the data to be used (compared to the traditional ANCOVA model), by also including the individuals who have no outcome measurement, but who do have a baseline measurement. Thus, the number of observations used by unstructured multivariate linear models is larger than the number of observations used by ANCOVA when missing outcome data are present. Moreover, unstructured multivariate linear models also deal with partially missing baseline measurements in RCT’s in the most statistically efficient way when the outcome is measured (34).
Unstructured multivariate linear models are advantageous because they can handle substantial drop out rates in RCT’s in an unbiased way under a ‘missing at random’ (MAR) assumption (36). In our study we used these models to compare VOCALISE at baseline and 12 months, under the assumption of missing at random (a weaker assumption than missing completely at random, which is used in the more traditional ANCOVA model). If data are missing at random, the emphasis is shifted so that, conditional on the fully observed variable (VOCALISE T0), the chance of seeing the partially observed variable (VOCALISE at 12 months) is assumed not to depend on the values of VOCALISE T0. Whether participants withdraw from the trial or remain, the distribution of their data is conditionally the same, because their unobserved future is based on their observed past (37). This approach is also preferable to using multiple imputation, a less efficient form of this type of analysis, since the two approaches broadly coincide, as the number of imputations gets large
Initially, mean total VOCALISE scores for the repeat participants, will be plotted at both time points to compare the scores of both groups (intervention and control). Then, we will compute an unstructured multivariate linear model to explore the impact of the DOORWAYS intervention at follow up on staff perceptions of barriers to change.
Unstructured multivariate linear model
In the model, the correlated outcome variable will be staff perceptions of barriers to change (VOCALISE) and the main predictors of interest (included as fixed effects) will be as follows:
Time: the adjusted change in the outcome between baseline and the 12-month follow-up.
Treatment group (intervention effect): this variable shows the adjusted change in score between groups (control and intervention) at follow up.
In our previous papers, we showed that ward climate and occupational status affected perceptions of barriers to change (18, 23). In this study, ward and occupational status will therefore be included as covariates in the model, as follows:
Considerations for model interpretation
There are some points to note before describing the results because the interpretation of unstructured multivariate linear models for the analysis of RCT data is different from the usual interpretation of linear models.
The intervention effect variable estimates the difference in group scores at follow up, adjusted for all other included covariates. If coded to provide estimates for those who participated in the intervention wards, it assumes an interaction between group and time because this variable comprises 2 groups: 1) those who were in the control group at follow up and 2) everybody else (baseline sample and those who did receive the intervention at follow up). When the coding is changed to provide estimates for those in the control group, there are then 2 groups: 1) those who were in the intervention group at follow up and 2) everybody else (baseline sample and those who were in the control group at follow up).
The models do not measure a main effect of time because as discussed above, an interaction between time and the intervention effect variable is assumed. The time variable allows an estimate of the adjusted change in the outcome between baseline and follow up. By changing the coding in the intervention effect variable, the estimates for the time variable are also restricted to the control group only or the intervention group only. And, because there is an interaction between group and time in the intervention effect variable, the effects within each treatment group are expected to be different over time.
The constant represents the estimated mean outcome score. As the models adjust for occupational status, this score is based on occupational status =0 (direct care); and the reference category for ward, which was ward 1, where staff had the most negative perceptions of barriers to change. The constant is the same whether the intervention effect variable is coded to represent those who did, or those who did not receive the intervention because of the coding (which enforces a 0 treatment effect at baseline in order to meet the assumptions of an RCT).
The estimates for ward and occupational status are the mean outcome score differences between the different categories of ward and occupational status across time, given the assumption that both arms of the trial started with the same scores at T0. Therefore, for example, the estimate for occupational status is the mean score difference between the two categories of occupational status, adjusting for ward, time, and the intervention effect that forces the mean scores to be the same at baseline. The estimates for ward and occupational status are across time, and are not changed by recoding the variable for intervention effect.
To aid interpretation of significant estimates, the mean scores of those included in the repeated measures sample will be compared to the mean scores of those included in the full dataset. This will also provide a sense check for the more complicated model results. A figure showing the mean score of the repeated measures sample will be compared to mean VOCALISE scores post hoc, which were calculated using the post estimation command lincom, in Stata 14. This command computes point estimates, standard errors, p-values, and confidence intervals for the linear combination. These are based on the model, which adjusts for baseline differences, and therefore both groups have the same baseline score.