We conducted a mixed methods, concurrent process evaluation[5,10] during the INFORM trial to assess intervention fidelity and experiences of participant teams. During all three intervention workshops, we collected data using attendance lists, intervention delivery checklists, participant team worksheets, exit surveys, and expert observations.
Data Collection
During the first workshop, teams completed a goal setting worksheet. They outlined their specific INFORM goal to increase care aide involvement in formal team communications about resident care, strategies for goal attainment and measures to provide feedback to teams on goal progress. At the second and third workshops, each team made a presentation about their activities and goal progress since the previous workshop. Study investigators with expertise in the core components of INFORM carried out structured observations of the presentations. At the end of each workshop, teams also completed an exit survey, and workshop facilitators completed an intervention delivery checklist indicating whether each workshop agenda item was delivered as planned.
Sample
This study includes 106 nursing home care units randomized to basic and enhanced assisted feedback arms. These 106 units are clustered in 33 different nursing homes (range of 1–10 units per home, median = 3).
Measures
Intervention fidelity is measured using 11 items (Table 1) that reflect fidelity delivery (4 items), receipt (4 items), and enactment (3 items). All 11 items show sufficient variation. Three authors (LG, MH, PN) reached consensus that these items reflect delivery, receipt, and enactment of the core components of INFORM described above. Because these items reflect different aspects of fidelity rather than a single fidelity construct, they are not scaled together.
Perceived value of the intervention. Four measures of team perception of intervention value are based on exit survey data from the three workshops. These measures include team perceptions for: (1) the value of workshop 1 material (average of 6 items; e.g. The preworkshop exercise was valuable, the presentation on SMART goals was valuable, alpha = .89), (2) the value of workshop 1 inter-team activities (average of 2 items; e.g. Discussions/feedback from other teams helped with setting performance goals, alpha = .64)[2], (3) the value of workshop 2 (average of 3 items, e.g. Creating the report back presentation was valuable, alpha = .86), (4) the value of workshop 3 (average of 3 items, e.g. Discussion period after report back was valuable, alpha = .81). All items used a 5-point agreement Likert scale.
Overall fidelity enactment reflects expert assessment of a team’s implementation of the core intervention components in the intended situation. We measured it with a single-item enactment rating scale (1 = very low enactment, with no/almost no activities undertaken to improve care aide involvement in formal team communications about resident care; 5 = very high enactment, with extensive activities undertaken). The rating was provided at the end of workshop 3 jointly by the two individuals who delivered all three INFORM workshops and who were most familiar with each team’s activities. Fidelity enactment is a binary variable, generated by recoding 1–3 as lower enactment and 4 and 5 as higher enactment.
Outcome: Care aide involvement in formal team communications about resident care is one of 10 concepts measured by the Alberta Context Tool, a comprehensively validated tool to assess modifiable features of the care unit work environment.[29] We used a modified score for formal team communications, asking care aides how often (in the last typical month) they participated in: (a) team meetings about residents, (b) family conferences, and (c) change-of-shift reports (each item rated from 1 = never to 5 = almost always). The modified score was generated by recoding each item (1 and 2 to 0, 3 to 0.5, 4 and 5 to 1) and summing recoded values (possible range: 0–3). To gather data on formal team communications, we administered the Alberta Context Tool by computer-assisted structured personal interview to a minimum of 10 care aides on each unit participating in the INFORM trial, at baseline (2 months before INFORM) and follow-up (2 months after the last support workshop).
Analysis
For study objective 1 we used descriptive statistics to examine the fidelity with which the INFORM intervention was delivered, received, and enacted. For study objective 2 we used chi-square and Fisher’s exact tests to examine whether intervention intensity (study arm) was associated with differences in fidelity. A Shapiro-Wilk test showed the Perceived Intervention Value variables to be non-normally distributed (p = .000 for all four variables). The Mann-Whitney U test was therefore used to examine whether the perceived value of the intervention workshops differed by study arm. For study objective 3 we used hierarchical mixed modelling (GLMM ML estimation, SAS), which accounts for clustering of units within facilities. This modelling examines the variance in INFORM’s primary outcome (care aide involvement in formal team communications about resident care) that is explained by each of the 11 fidelity delivery, receipt, and enactment items. The posttest score was the dependent variable with the baseline score entered as a covariate. Lastly, we conducted repeated measures analysis of variance to examine whether the relationship between time (baseline and follow-up) and care aide involvement in formal team communications about resident care was moderated by overall fidelity enactment. In other words, did improvement in care aide involvement over the study period differ for low and high enactment teams? Our mixed models suggested that the variance explained by facility clustering was small and statistically non-significant (facility-level random intercept = 0.0002, p = 0.3733, intra-cluster correlation = 0.0411). Therefore, we did not include a random facility-level intercept in our repeated measures model.
[2] 8-items used to measure the value of workshop 1 were factor analyzed (EFA with oblimin rotation) and loaded on these two factors: value of the workshop material and value of the inter-team activities. The value of workshops 2 and 3 was assessed using three items each so were not factor analyzed.