Study design
The embedded-explanatory mixed methods design [21] process evaluation will be an integral part of the Workwell trial. Investigations will draw on the Logic Model developed for the intervention (Fig. 2). We will use the CFIF to examine fidelity to Workwell [18]. Fidelity acts as a moderator between interventions and intended outcomes. Its evaluation allows for investigating whether any positive outcomes could be improved on, and whether negative outcomes are due to poor implementation or issues with the intervention. The process evaluation will also be guided by the NPT [20]. This will influence the structure of interview topic guides with participants and therapists, as well as participants’ employers and therapy line managers, support the interpretation of the Thematic Analyses of interviews and aid synthesis of data across data collection methods [22]. NPT facilitates understanding the perspectives of both therapists delivering JRVR in practice and participants embedding JRVR interventions into their daily lives. The process evaluation team include members of the research team who developed the Workwell JRVR, therapist training and trial design, as well as those involved in trial management, but not analysis of the trial outcomes.
To assist future comparative evaluations of VR trials, the process evaluation protocol was planned to be similar to that within the RETAKE trial, evaluating Early Stroke Specialist Vocational Rehabilitation to enable people in returning to work [23].
Study participants
Participants include: trial participants in the intervention group who received Workwell JRVR, either continuing in employment or no longer employed at 12-month follow-up; trial participants in the control group; line managers or employers of participants receiving Workwell JRVR; Workwell-trained occupational therapists; and therapy line managers of participating therapists.
Informed consent
All participants will be provided with an information sheet and opportunity to ask questions. Taking part in the process evaluation will be optional. Written consent to participate in semi-structured interviews, surveys, questionnaires or focus group (as relevant) will be received from participants. This includes consent for audio-recording interviews, focus groups, and the initial JRVR treatment session for one participant each therapist treats.
Patient and Public Involvement
Patient and Public Involvement and Engagement (PPIE) were ensured in all stages of the trial through the trial Patient Research Partners Group (PPRG), of three members. A member of the PPRG is a co-applicant on the grant, and assisted in identifying research questions, designing the study and trial protocol, and is also a member of the Trial Management Group and Trial Steering Committee. All three members meet regularly and have assisted with: review of patient facing materials, including interview topic guides; advising on communication with participants; and planning and conducting the PPI participant interviews (see later). The PPI group will be involved in discussion of trial results, data analysis and interpretation of process evaluation findings and presentation of results.
Data collection
Table 2 indicates the relationship between research aims, questions, data sources and methods. Data sources are described below. Data collection or extraction will be conducted by members of the process evaluation team.
a. Measuring Fidelity
A range of predominantly quantitative data collection methods will be used informed by the CFIF (Table 3).
Workwell therapists’ ability and confidence to deliver treatment
Each participating Workwell therapist will complete a questionnaire before and shortly after the training programme about: their knowledge of and confidence in delivering components of Workwell JRVR, and views about delivering evidence-based practice [24]. Following therapists’ completion of the mock telephone WES-RC and treatment plan (as part of training), the Workwell training team will use a checklist to record each therapist’s ability to complete components appropriately, with feedback provided to therapists. Mentoring checklists and associated recommended action points for therapists will also be reviewed to explore therapists’ abilities when assessing and developing treatment plans for their first participant.
Treatment Records – Workwell frequency, duration, coverage
Workwell therapists complete Treatment Records Part 1 (at start) and Part 2 (at discharge) for the trial participants they treat. These summarise: if treatment started within four weeks; numbers of treatment sessions attended and duration; work site visit and duration (if occurred); and telephone review and duration. From this frequency, duration and time scale of treatment can be derived. Records also include: if participants did not attend or were unable to attend any appointments; reasons for discontinuation (if occurred); mode of treatment delivery; any other treatment provided; participant-reported adverse events; and any participant-reported benefits of Workwell JRVR.
WES-RC and Treatment Notes – Workwell content
Treating therapists complete the WES-RC and treatment notes for each trial participant they treat. This includes the participant’s: individual work barriers, three priority problems with work and JRVR solutions suggested; whether solutions were recorded in notes as provided and implemented; and duration of JRVR components (direct and indirect).
Intervention fidelity
A structured checklist will be used to assess fidelity of intervention delivery. Normally, treatment for the second participant of each therapist will be assessed, as this allows therapists’ time to have gained experience in Workwell JRVR delivery. Assessment will be conducted by members of the Workwell training team. Data will be obtained from: the audio-recording of the participant’s initial treatment session (i.e., WES-RC interview, joint problem prioritisation and initial treatment planning); their completed WES-RC assessment form, Workwell treatment notes and Treatment Record Parts 1 and 2. The checklist will be used to identify that: a) the WES-RC interview was conducted appropriately; b) priority problems were appropriately established with the participant; c) treatment content was appropriate for the agreed problem priorities of the participant; d) treatment content was delivered as planned within an appropriate time scale; and e) the final report for the participant reflected the treatment provided.
Other work provision, satisfaction with JRVR and usual care
Additionally, in the participants’ six-month follow-up questionnaire, we will include items to identify if intervention and control participants received: work advice, source(s) of this (written, health professional, employer-based or other) and content. We will also identify if they report receiving and reading the work self-help information pack and their satisfaction with work advice received in the Workwell trial. Participant-reported health, social and work resource use is also collected at six- and 12-month follow-up.
Fidelity data will also be collected during therapist interviews (see below).
b. Social and structural context
A range of predominantly qualitative data collection methods will be used. All interviews will be conducted using a topic guide informed by NPT, except the PPI interview. Examples of question topics and how they relate to the four NPT constructs are shown in Additional Files 2 and 3.
Pre-trial site survey and therapist interviews about usual VR service provision
Prior to Workwell provision, a survey will be completed by the lead therapist at each site to identify what work advice or JRVR is normally provided to patients with IA. Data collected will include: the numbers of patients provided with work advice each month; average duration of advice given; and a brief description of what this consists of. Following the site training visit, the lead Workwell therapist at each site will be interviewed about their therapy and rheumatology services’ current JRVR provision. This will be a short (15 minutes) telephone interview, including their views at the outset on whether Workwell JRVR could be implemented in future, potential barriers, and enhancers.
Questionnaires and interviews with therapists about Workwell training.
Each participating Workwell therapist will complete a questionnaire shortly after Workwell training asking about the relevance of each component of the training programme. Following completion of delivering Workwell JRVR at their site, therapists (one from each site n=18; or until data saturation is reached) will take part in a semi-structured telephone interview. This will include asking about their views of the Workwell training programme. There will also be an opportunity for therapists to take part in a focus group exploring future methods of implementing Workwell training, including using online training. This will build on work in the feasibility study, in which therapists provided views on training received and made recommendations for changes, now made in this trial [15, 25].
Interviews and focus groups with therapists about delivery and implementation of Workwell
The therapist semi-structured telephone interviews (see above) will then explore therapists’ views about delivering Workwell JRVR during the trial, fidelity of delivery, future implementation in practice and whether their usual work advice service changed during the trial.
Therapy service managers’ semi-structured interviews
Therapists at each site will identify their appropriate therapy line manager to be contacted for consent for a 10 to 15- minute telephone interview (one from each site n=18, or until data saturation is reached). Managers’ views will be explored about Workwell JRVR implementation during the trial, and potential future implementation in clinical services.
Interviews with trial participants
Semi-structured interviews will explore trial participants’ views about Workwell JRVR. This will focus on identifying which components of the Workwell intervention participants implemented, whether/ how their job changed as a result of the Workwell intervention, which components they consider most or least helpful to assist them staying in work (e.g., job modifications, flexible hours, self-management), whether JRVR enabled them to make changes (if any), and what contextual factors helped them to stay in work. If participants are no longer working, contextual factors contributing to them stopping work and their views of the JRVR received will be explored. Trial participants’ acceptability of the intervention and its provision within the NHS will also be explored.
One-to-one telephone interviews will be sequentially completed after participants have completed their 12-month follow-up questionnaire. This will ensure that they have completed JRVR, and sufficient time has passed for any changes to have an effect (if any) on their work. Interviews will be undertaken with purposefully selected participants in the intervention group who consent to being contacted for interview. Participants will be selected based on those who are in work (n=15) and not in work (if any, up to n=10), reflecting the gender distribution of IA (two-thirds women) and across four job skill level groups, with three to four each from: Level 1 (elementary occupations); Level 2 (administrative, caring, leisure, sales, customer service; process, plant and machinery operatives); Level 3 (associated professional and technical/ skilled trades); and Level 4 (professional and managerial) [ONS, 2010]. Participants’ job skill level groups are identified from their job titles in their baseline questionnaires. Interviews will usually last 45 to 60 minutes.
Additionally, members of the PPI group will also conduct semi-structured telephone interviews (10 to 15 minutes) with a convenience sample of participants from both the intervention and usual care arms of the trial, who consent to being contacted for interview after the 12-month follow-up questionnaire is completed. The interview topic guide was developed by the PPI group and differs from above as it focuses on trial procedures, the patient facing documentation, as well as trial participants’ views implementation of Workwell JRVR and the self-help information pack in the NHS.
Interviews or surveys with participants’ line manager/ employer
A selection of intervention group participants’ line manager or employer (n=10) will be interviewed or surveyed about their views of the Workwell JRVR received by their employee. Following completing the 12-month follow-up questionnaire, participants will be provided information about the line manager/ employer interview or survey to consider. For those interested, they are asked to discuss the study with their line manager/ employer. They will be provided with a coaching script as a guide to help them explain this part of the study (if they wish to use it). If their line manager/ employer agrees to take part, a member of the process evaluation team will receive their documented verbal consent at the agreed time/date of their interview and then conduct the telephone semi-structured interview (10 minutes). We also have the option for line managers/ employers to complete a short survey by e-mail, if they do not have time to, or prefer not to, take part in an interview. Any surveys returned can be considered as providing consent. It is highly likely that participants who have already disclosed their arthritis to their line manager/ employer will agree to employer contact. For those participants not interested in employer contact, we will ask if they are willing to indicate why not. We will explain there are many reasons why people prefer not to have this contact and that their response will help us to understand these.
Identifying potential contaminants
During therapist interviews, we will ask if participants in either group received more work-related intervention than planned in the trial (see Additional File 3). Additionally, we will use six-month questionnaire data (about other work services accessed) and patient-reported resource use data (changes to workplace data) to identify whether participants in the treatment or control groups received work interventions from elsewhere, and whether this differed between groups.
Data analysis
The process evaluation team will analyse data collected and support the PPI group in analysing the PPI interviews, if required. Quantitative data will mainly be analysed and presented using descriptive statistics. Interviews and focus groups will be digitally audio-recorded and transcribed verbatim, with names replaced by codes or pseudonyms. Qualitative data will be managed using the NVivo 12 software. Qualitative analysis will be done inductively based on Braun and Clarke’s thematic analysis method [22]. Then, the codes will be organised into themes and mapped under the four NPT constructs. Pseudonyms will be used where participants’ accounts are directly quoted.
To ensure internal validity and reliability, the following strategies will be employed: each transcript will be checked against the recording to ensure no mistakes during transcription; validity and reliability of the emerging themes will be supported by asking two researchers to analyse the data independently and agree themes; one other member of the team will then independently review two interview transcripts with participants and two with therapists and their analyses [26]; and through regular discussion of the themes and definitions with other members of the process evaluation team and PPRG. The final relevant components of the report (e.g., participant interview report to participants) will be returned to interviewees to confirm it reflects their experiences [22, 26].
Workwell JRVR adherence will be calculated using data from the participants’ Treatment Records completed by Workwell therapists. Descriptive data will be extracted on frequency, duration, intensity, and dose of JRVR. If Workwell was not attended or discontinued, reasons will be extracted if recorded. To identify Workwell content, the following will be analysed: WES-RC, treatment notes and Treatment Log (i.e., coded content of the treatment provided, identifying what types of VR (direct and indirect) were provided, e.g., conducting the WES-RC interview, providing VR, using resources to identify solutions, writing reports). The numbers of work site visits and employer contacts will also be recorded. From the WES-RC we will descriptively analyse: frequency of health issues reported as affecting work (e.g., pain, fatigue, stress); type of work (categorised into job skills levels 1 and 2, or 3 and 4); work barriers identified, and priority problems identified. The WES-RC therapist notes will be content analysed to identify the VR solutions provided and whether solutions were reported by patients as implemented [27]. This data will be mapped on to the Template for Intervention Description and Replication framework (TIDieR) to describe the Workwell intervention delivered [28]. We will also use TIDieR to analyse data collected from the six-month follow-up questionnaire to describe usual care received by both intervention and control groups.
The audio-recording of each therapist’s initial treatment session with one participant (up to n=40) will be listened to by members of the research team, using a structured fidelity checklist, to assess whether the therapist went through the assessment process appropriately, identified and appropriately prioritized the participant’s problems, planned, and commenced an appropriate treatment plan. The completed WES-RC and the accompanying trial treatment notes for that participant will also be analyzed to evaluate whether the problems, plan and solutions, as discussed, were recorded in the WES-RC and the treatment notes record the fulfillment of the treatment plan [29-31].
Therapist training questionnaire data will be analysed descriptively using medians (IQRs) to explore views about components of the training. Inferential statistics will be used to investigate if there is any self-perceived change pre- to post-training in knowledge of and confidence in delivering Workwell JRVR and adopting new interventions into practice. We will also content analyse: comments in the post-training questionnaire made about the training duration and content, and the mock telephone WES-RC interview and mentoring checklists, to further understand therapists’ views about training and ability to deliver Workwell.
Data synthesis
The analysis of the different elements of the process evaluation will be conducted prior to the main trial findings being available. Each component will be reported separately. The process evaluation team will then review and integrate the components. Investigation of context, implementation and mechanisms of impact will be guided by both CFIF [18] and NPT [19], with reference to the Workwell logic model. NPT provides structure to process evaluations by focusing attention on the range of actors, times and places involved in implementing interventions [[20]. It will also aid data synthesis from multiple sources (participants, participants’ line managers/ employers, therapists, service managers) and will provide a framework to assist understanding the mechanisms of actions of JRVR, if the trial is successful. Emerging findings from the different components of the process evaluation will be discussed amongst the team and with the wider trial team to facilitate transparency.
The process evaluation data will be analysed independently from the Workwell trial outcome evaluation. The analyses are conducted by two separate teams. Once the process evaluation and Workwell trial data analyses are complete, the process evaluation findings will be reviewed in the light of trial findings. Reviewing both analyses can aid understanding why the intervention or different components were successfully implemented or not, potential mechanisms of impact and explain trial outcomes. We will discuss the findings with therapists to further explore issues around service implementation, including identifying people with JRVR needs and methods of delivering training. The findings from the trial participant and PPI interviews will especially aid us in understanding what working people with arthritis consider applicable ways to implement JRVR.
The Workwell trial has been impacted by the COVID-19 pandemic. Half-way through the treatment period, the trial had to pause for between four to eight months across different sites. At re-start therapists switched to remote Workwell JRVR delivery via telephone or videoconferencing, whilst working in stretched circumstances. Many participants were experiencing increased job, personal and health stresses, working from home and were unused to remote treatment. We will also therefore explore participants’ and therapists’ views of face-to-face versus remote JRVR delivery, albeit in the context of difficult circumstances.