In the FRESH trial (11), 78 traumatic brain injured participants were randomly allocated to the VR intervention in addition to their usual NHS rehabilitation (intervention group) or usual NHS rehabilitation alone (control group) in three regions (the North West, London, and Yorkshire) of England over 12 months. The primary outcome was participants’ work status, defined as a minimum of an hour per week of paid or unpaid work, analysed at 12-months using an intention-to-treat approach.
Implementation fidelity was measured as part of an embedded process evaluation. Data collection was longitudinal (Additional file 1). Quantitative process data consisted of content of treatment records, fidelity checklists, mentoring records and clinical occupational therapy records). Qualitative data collection methods included interviews, therapy and mentoring records. Both types of data enabled evaluation of the intervention process implementation and its fidelity during the study as subsequently recommended by Toomey et al (2017) (40).
The Conceptual Framework for Implementation Fidelity (CFIF) (30) was used to guide both the measurement of fidelity and understand factors affecting its delivery. The CFIF structure facilitated the development of measurement tools, including a fidelity checklist (12, 41–43), data collection (44) and analysis (45).
Participants included patients recruited to the FRESH feasibility randomised controlled trial (RCT) who were randomised to receive the FRESH VR intervention. Inclusion criteria were people aged 16 years or above, admitted to one of three major trauma centres for 48 hours or more, with a new TBI (within 8 weeks) and who were in paid or unpaid work or full-time education prior to injury. Full eligibility criteria in the FRESH RCT are explained elsewhere (46). Clinical records and intervention session case report forms (CRFs) were collected for every intervention participant (n = 38). Purposive sampling was used to identify and recruit five participants for telephone interviews from each site with a range of demographics and TBI severity who had received the intervention (n = 15).
The five OTs who delivered the intervention were Health and Care Professions Council registered with expertise in VR. OTs attended two days of training, plus an additional day six-months after intervention delivery commenced. This is described elsewhere (25, 26). Training was delivered by a team of four OTs with expertise in VR, TBI and research. Training was supplemented by an intervention manual and monthly individual mentoring by a member of the training team to support implementation during the intervention delivery period. OTs could contact their mentor for advice when they needed it.
Employer participants included line managers, human resource professionals, or occupational health professionals of patient participants in employment or teaching staff linked to participants in full-time education. A convenience sample of 15 employers were recruited (five from each site). Only employers of people with TBI (PwTBI) randomised to receive the intervention were eligible.
NHS staff participants included those staff at each site, who, in their usual role, were involved in managing, commissioning, or delivering TBI rehabilitation. A convenience sample of 15 NHS staff were recruited (five from each site).
The VR intervention, described using the TIDieR checklist in Additional file 2 and elsewhere (47–49) was delivered to the FRESH participants by an OT (described above). The primary focus of the intervention was preventing job loss and optimising employment and education outcomes. The intervention started within eight weeks of injury and lasted up to 12 months. The logic model for the FRESH intervention is described in Additional file 3.
Ethical approval was granted by Integrated Research Approval System (REC Ref: 13/EM/0353 and the University of Nottingham Ethics Ref: D14112013 FRESH). The process for obtaining participant (patient participants, OT participants, employer participants, NHS staff participants) informed consent was in accordance with Research Ethics Committee guidance and Good Clinical Practice (50).
Data was collected about every session delivered by every OT during the FRESH intervention delivery phase between January 2014 and January 2016. A description of each data collection tool, time points for data collection, data type, which CFIF construct they related to, their purpose and data usage are shown in Table 1.
Table 1
Data Collection Tools and Timeframes for Data Collection
Tool | Timeframe | Data type | CFIF construct | Tool purpose | Data usage |
Intervention CRF | Continuous | Quantitative | Adherence | Quantity of components (10-minute units) delivered per session, direct participant activity (face-to-face, telephone), travel, indirect activity (e.g. session preparation) | Triangulated with clinical record |
Clinical record | Continuous | Quantitative and qualitative | Adherence and moderating factors | Description of intervention plus evidence of correspondence | Triangulated with intervention CRF |
Fidelity checklist | Quarterly at monitoring visits | Quantitative and qualitative | Adherence and moderating factors | Oversight per OT, extent of components delivered (always to never), moderating factors affecting delivery. Advise OTs | Triangulated with intervention CRF, clinical record, mentoring CRF. |
Mentoring CRF | Monthly | Qualitative | Adherence and moderating factors | Intervention summary provided by each OT, details factors affecting delivery and potential solutions | Triangulated with intervention CRF, clinical record, fidelity checklist |
Interview with OTs | Start of intervention delivery & End of delivery | Qualitative | Moderating factors | Addressed acceptability of the intervention, the factors affecting delivery and solutions to overcome barriers | Triangulated with mentor CRF, clinical record. |
Interview with PWTBI, their employers and NHS staff | End of intervention delivery | Qualitative | Moderating factors | Addressed acceptability of the intervention, the factors affecting delivery and solutions to overcome barriers | Triangulated with mentor CRF, clinical record. |
Key: Continuous = OT completed the CRF and clinical record after each session. CFIF Adherence includes intervention content, coverage, frequency and duration of intervention; CFIF Moderating factors includes participant responsiveness, intervention complexity, strategies to facilitate implementation, quality of delivery, recruitment, and context. |
Quantitative adherence data (content, coverage, frequency and duration of intervention); and qualitative data explaining moderators of fidelity (participant responsiveness, intervention complexity, strategies to facilitate implementation, quality of delivery, recruitment, and context) were extracted from the tools and compared, through triangulation, for data verification and to identify missing data by an independent researcher. Each intervention session was recorded on an intervention CRF that was modified from one initially developed by Phillips et al (48) plus clinical records following the OTs’ own local policies and procedures.
A fidelity visit checklist (Additional file 4) for use by an expert rater, was developed to measure whether OTs adhered to the core intervention processes. It was based on an observational checklist designed by Hasson et al (44) informed by the theoretical constructs of CFIF. As recommended by others (17, 27), the checklist comprised the intervention components and core processes extracted from the FRESH intervention logic model (Additional file 3). Fidelity to each was rated on a 5-point ordinal scale as delivered: ‘always’, ‘often’, ‘sometimes’, ‘seldom’ or ‘never’ (where ‘always’ scored 1 and ‘never’ scored 5). Factors that affected the delivery of the intervention were recorded alongside. Guidance notes helped assignment of ratings.
Each OT received four fidelity check visits by a post-doctoral research OT (JP) who was also a trial mentor. OTs provided anonymised copies of clinical notes and intervention CRFs prior to a visit. The researcher and OT met and discussed the intervention delivered to participants. The researcher systematically rated each component on the checklist. Following the visit, data were recorded on an Excel document by a member of the study team. Fidelity checklists were discussed between the mentoring team and were used to identify non-adherence to intervention delivery and then translated into topics for skill-building during ongoing mentor sessions (51).
Content of each monthly mentoring session was recorded on a mentoring CRF by the mentor, additional mentor support by email and phone calls were also collected.
Interviews with OTs were conducted early after training and later to capture the OTs varying experience of delivering the intervention. PwTBI and their employers and NHS staff were interviewed at the end of the intervention. Interviews followed a topic guide informed by the theoretical constructs of CFIF (30, 44) to capture qualitative data on factors affecting implementation fidelity. Interviews took place by telephone and lasted approximately 45 minutes. They were digitally recorded, fully transcribed, cleaned and the data was uploaded to SQR Nvivo software for analysis.
Data Analyses
All constructs of fidelity, based on the CFIF, were measured to avoid gaps in reporting and data were compared to:
The intervention logic model and a benchmark were used to guide data analysis and interpretation. There is no specific guidance to answer, “How much variation is allowed in fidelity measurement?” but Durlak and DuPre (52) (2008) indicated, in their meta-analysis of 542 interventions between 1976 and 2006, that outcomes were effective when interventions were delivered with 60–80% fidelity. They advised researchers not to expect 100% fidelity and recommended that the variation in fidelity across sites should be reported rather than only presented as a mean, because this can mask expected variation.
Carroll et al (30) suggest using a benchmark against which to measure fidelity, as this adds to understanding the quality of the intervention delivered. The benchmark used for this comparison was Phillips’ (47) description of an early VR intervention for PwTBI, which informed the development of the FRESH intervention. Quantitative data about the proportion of components delivered by the OTs were compared to the same data provided by Phillips (2013) to illustrate how closely they matched.
Fidelity checklists were analysed using the 5-point scale after each monitoring visit. Data obtained from clinical records and intervention CRFs were triangulated to identify variations in fidelity and disagreement between data sources. Where there was disagreement e.g. in the recording of a session, the clinical record was considered more likely to represent what had occurred because therapists were more familiar with this form of documentation than the intervention CRF. Descriptive statistics were used to describe the quantity and content of the intervention delivered.
Intervention content was analysed by comparing each CRF with the description of the intervention session in the clinical records. The proportion of time spent on each intervention component was calculated from the CRF. Duplicated data were removed from the analysis and missing data were recorded.
The frequency and duration of the intervention was calculated using the dates recorded in the CRFs and clinical records. Total time spent in direct contact and indirect contact with patients was taken from the content proformas. Data from the CRF and clinical records were compared to these timeframes.
Text describing factors moderating implementation fidelity (participant responsiveness, intervention complexity, facilitation strategies and quality of delivery) were extracted and triangulated across multiple records (fidelity visit checklists, clinical records, mentoring CRFs and interviews). Interview transcripts were analysed by at least two researchers using the framework method (53).