A mixed methods implementation study grounded in the CFIR was conducted to evaluate the relationships between implementation determinants and outcomes in the 2019-2020 cycle of SWITCH. Evaluation approaches followed recommended data collection and analytic methodologies of CFIR, developed by Damschroder and colleagues (22, 26). To our knowledge, this is the first documented adaptation of the CFIR mixed methods protocols with the goal of understanding relationships between implementation determinants and outcomes within a school health promotion context.
Participants and procedures
In the 2019-2020 iteration of SWITCH, 52 schools were enrolled, 30 (57%) of whom had already taken part in the program in previous years. Demographic information for these schools is shown in Table 1. The cyclical training (fall) and implementation (spring) process of SWITCH across the academic year facilitates a continuous quality improvement process (31), whereby feedback from schools and implementation outcome data drive modifications to the program each year. More information about the training process can be found in Additional File 1, our previously published article (11), and the program website (https://www.iowaswitch.org/). Briefly, schools were asked to form a wellness team which comprised three members of staff across different school settings (e.g., classroom teachers, physical education, food service, other teachers, administration, counselors, nurses, etc.) and to register prior to the beginning of the academic year. Following registration, schools were asked to attend a total of four webinars and an in-person conference during the fall semester, as well as complete several pre-program audit tools. The implementation phase spanned a 12-week period from January–April of 2020, but due to the coronavirus (COVID-19) outbreak, schools were forced to close in Iowa on March 13th thus forcing a transition to virtual communications/implementation after week 8 of the program. It was not possible to capture final outcome data, but schools completed the midpoint evaluation of school implementation. Below we outline data sources for implementation outcomes and determinants, and the steps taken to rigorously analyze these data.
Measurement of Implementation Outcomes: Adoption, Fidelity, and Penetration
The field of D&I offers many frameworks and theories to help researchers and practitioners discern why evidence-based practices are or are not implemented in routine care. Regarding implementation outcomes frameworks, the framework by Proctor and colleagues (32) conceptualized several distinctive outcomes that are important to include within implementation evaluations: 1) acceptability (the degree to which an innovation is a perceived good fit), 2) adoption (intent to implement), 3) appropriateness (degree of compatibility within setting), 4) cost (to implement, value for money), 5) feasibility (possibility of successful implementation), 6) fidelity/compliance (executed as intended), 7) penetration (reach within setting), and 8) sustainability (long-term impact). For the purpose of this study, we chose to examine the determinants of adoption, fidelity, and penetration among schools enrolled in SWITCH due to the heavily integrated implementation practices needed to create systems change in the school setting.
Adoption is operationalized by Proctor and colleagues (32) as “intention, initial decision, or action to try or employ an innovation or evidence-based practice” (p. 69). Thus, we measured adoption through implementation surveys at the six-week mark, examining uptake of best practices in various settings (use of curricular modules, posters, reinforced themes through discussion and tracking). Each best practice was scored as 0 (not at all implemented), 2 (somewhat implemented), and 3 (fully implemented) and a summed score was generated based on the average of each component, to give possible range of 0–9.
Fidelity relates to “the degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the program developers” (p. 69) (32). The quality elements of SWITCH comprise, wellness team meeting (ideally at least once per week), using SWITCH website to promote student behavior tracking, engaging parents and other stakeholders, and integration of SWITCH modules/posters across the school setting. Fidelity therefore was calculated by using a summed score of quality elements which were scored the same way as best practices, giving a possible range of 0–12.
Finally, penetration is defined as the “integration of a practice within a service setting and its subsystems.” (p.70) (32) this was calculated by determining the number of participants who used or interacted with an evidence-based practice, divided by the total number of participants eligible or within the sample. Since the behavioral tracking and goal setting interface is an integral component for students (12), it provides a good indicator of how many students are actively engaged in SWITCH within each school, thus providing data on penetration. We used data from SWITCH behavior tracking across weeks 1–8 (to account for COVID-19-related school closures). These data are presented as a decimal score (range 0–1.0, translated to 0–100%).
To assess baseline readiness for implementation, the School Wellness Readiness Assessment (SWRA) tool (21) was previously developed in line with the theory of organizational readiness for change (33, 34) and community capacity-building frameworks (35). The SWRA applies these concepts to school environments, taking into account the unique, complex structure and specific settings within schools that impact student health, including classrooms, physical education, and lunchroom settings, and the broader school leadership and cultural context.
The SWRA includes questions across four subscales designed to assess setting-specific and school-wide wellness readiness: classroom readiness, physical education (PE) readiness, food services readiness, and school readiness. The SWRA items were assessed using a 5-point scale (strongly disagree, disagree, neither agree nor disagree, agree, and strongly agree scale, coded as 0, 1, 2, 3, and 4, respectively). A copy of the SWRA is provided in Additional File 2. Wellness teams completed the 40-item SWRA through the program website. Scores for each of the subscales were calculated by averaging together the item responses in each section with higher scores representing higher states of readiness in specific settings and schools.
Qualitative Interviews Grounded in CFIR
Following procedures developed by Damschroder and colleagues (26, 27, 36), an interview guide was developed which aimed to understand the influence of each CFIR domain on implementation of SWITCH (see Additional File 3). Questions were open-ended, examples included, “What is your perception of the quality of the modules, posters, and other SWITCH materials that were provided?” (Innovation Characteristics – Design Quality and Packaging) and “How do you think your school culture affected the implementation of SWITCH programming?” (Inner Setting – Implementation Climate). Interviews were conducted by a qualitative and survey methodologist to ensure impartiality in responses from school wellness teams. Participants were encouraged to respond openly and candidly about their experiences with implementation and did not set a time limit on these conversations to ensure in-depth understanding of each context and implementation climate. Interviews lasted between 31 and 63 minutes, conducted through video conferencing software (i.e., Zoom), and transcribed verbatim.
Qualitative data coding and case memos
The structure of the interview guide facilitated a predominantly deductive data analysis approach, in that each of the questions corresponded to a construct within each of the framework domains (26). However, we remained open, such that any themes that emerged through inductive approaches were included in our analyses (37). First, the lead and second author met to develop a coding consensus document (Additional File 4), which described each CFIR construct and anticipated potential responses and themes that would emerge through the data. Applying the CFIR systematic coding approach facilitated the assignment of numerical scoring to the qualitative data, such that if a particular construct was deemed to have a positive influence on implementation based on interview responses, a score of +1 or +2 was assigned for that construct. Conversely, if a construct was deemed to be a negative influence, a score of -1 or -2 was given. If it was not clear whether a positive/negative influence manifested, a score of 0 was given, a score of “X” was used for mixed results (later coded as 0 for quantitative analysis, see Additional File 5 for details on CFIR rating rules) (26).
Second, to establish inter-rater reliability, the two coders selected five transcripts and created independent case memos using the CFIR memo templates (36). Scores were compared and a percent agreement score was calculated, if the overall agreement score was <80%, the coders met to ensure consensus before coding another set of five transcripts. Once ≥80% agreement was met, the second author coded the remaining transcripts, before a randomly selected set of 5 transcripts was reviewed by the lead author. All coding was completed in memo documents (see Additional File 6). Finally, to facilitate content analysis and interpretation of trends in interview data, all memos were entered in to NVivo qualitative analysis software and coded into respective nodes, following the CFIR codebook template (36).
All school demographic, implementation outcome, and implementation determinant data were merged using SAS software (Version 9.4, Cary NC) to facilitate descriptive and inferential analyses. First, descriptive tests were conducted to obtain means (and SD) for all implementation outcome and determinant data, then split by experience level (0= inexperienced, 1= experienced). Independent samples t-tests were conducted to understand differences in implementation outcomes according to experience level. Pearson bivariate correlations were run to establish correlations between implementation outcomes and determinants to examine associations and to understand potential influences of implementation for schools that experienced greater success. All tests were run in SAS software, and α significance was assumed as p<.05, correlations with p<.10 were also highlighted due to the novel nature of this work. Following inferential testing, the research team explored qualitative extracts using NVivo as a means to contextualize findings from correlation analyses. Such an approach allowed for deeper contextual understanding of implementation practices which triangulate implementation determinants and outcomes (11).