Use of an activity-based time tracking tool to support implementation of a school district-level technical assistance intervention

doi:10.21203/rs.3.rs-4707882/v1

Download PDF

Research Article

Use of an activity-based time tracking tool to support implementation of a school district-level technical assistance intervention

https://doi.org/10.21203/rs.3.rs-4707882/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background. Detailed time and cost data are often lacking in implementation science, particularly in school-based interventions. In a pilot intervention in one Chicago Public Schools’ geographic network, a Network Specialist was hired to provide schools with tailored technical assistance (TA) to support compliance with over 50 health-related policies (the Healthy CPS initiative). This study describes the methods for developing and implementing an activity-based time tracking tool to assess the Network Specialist’s fidelity, time, and cost in providing TA using a Multi-Tiered System of Supports framework (“Tier 1” universal support, “Tier 2” targeted support, and “Tier 3” intensive, individualized support).

Methods. The tool was developed in close collaboration with the Network Specialist to capture the Specialist’s interactions with schools between 2020–2023. Key informant interviews and asynchronous post-hoc feedback were qualitatively analyzed to assess the Specialist’s feedback on the tool. Descriptive statistics on school interactions, tiers of support provided, and domains of implementation support provided using the SISTER implementation science framework were generated from the tracking tool data. Differences in mean baseline Healthy CPS policy compliance based on the extent of schools’ interactions with the Specialist in each tier of support were computed using Wald tests.

Results. The Specialist described the tracking tool as feasible and useful in providing tailored support and advocated for its continued use as the intervention is expanded to additional networks. The Specialist spent the highest proportion of time and costs (41.4%, $39,117) providing intensive Tier 3 supports, and those supports were targeted toward schools with the most need. Schools receiving the most Tier 3 supports had lower baseline Healthy CPS compliance of 64.4%, versus 78.4% and 67.3% for schools receiving low and medium levels of Tier 3 supports, respectively (p-value = 0.045).

Conclusions. Expanded use of time and cost tracking is needed in implementation science, particularly for school-based interventions. Time tracking tools help collect data on intervention activities that inform decision-makers about how to implement interventions with fidelity. Our findings point to the value of using a collaborative, partner-engaged approach to developing the tracking tool with the end user to maximize its feasibility, usefulness, utilization, and sustainability.

Activity-based time tracking

school health and wellness policy

implementation science

costs

community-engaged research

Few interventions assess the time and costs associated with implementation strategies used and activities performed, even more so in evaluating school-based interventions.
The activity-based time tracking tool used in this study to support school-level health and wellness policy implementation revealed that the intervention targeted the most intensive supports to schools with the greatest need.
Time tracking tools should be developed in close collaboration with the end user to enhance their feasibility and usefulness.

In the United States (U.S.), schools are required to comply with a multitude of health and wellness-related policies at the federal, state, and local (school district) levels (1, 2). Keeping track of all of the policies, much less implementing them, is complex and there is no one-size-fits-all approach. In fact, Cook et al.’s School Implementation Strategies, Translating ERIC Resources (SISTER) framework identified 79 strategies that could be used to support implementation of school-based interventions grouped across nine domains: (a) use evaluative and iterative strategies, (b) provide interactive assistance, (c) adapt and tailor to context, (d) develop stakeholder interrelationships, (e) train and educate stakeholders, (f) support educators, (g) engage consumers, (h) use financial strategies, and (i) change infrastructure (3).

Chicago Public Schools (CPS), one of the largest school districts in the U.S., struggled to support schools with implementation of over 50 federal, state, and local health and wellness-related policies. Recognizing this, in 2016, CPS created the Healthy CPS initiative to help schools navigate and implement these policies (4). For the first four years of the Healthy CPS initiative, the district’s Office of Student Health and Wellness (OSHW) tried to support schools through standardized technical assistance (TA). However, it became clear that with over 600 schools (5), centralized TA was not realistic and did not provide schools with the level of support they required. Instead, CPS’ OSHW wanted to pilot test providing more localized support to schools through their 17 geographic networks; each network is comparable to a small-medium sized school district elsewhere in the country, with an average of more than 14,000 students per district and ~ 15–35 schools per network (6, 7). Each network is led by a Network Chief and each network provides administrative support, strategic direction, and leadership development to schools within the network (6).

When CPS schools need specific supports beyond the network, they are typically referred to offices and units throughout the district (e.g., Children and Family Benefits, OSHW, etc.). However, in 2020, in a collaborative community-academic partnership with the Policy, Practice and Prevention Research Center at the University of Illinois Chicago, CPS piloted an intervention to support one network in the west side of Chicago with implementing Healthy CPS. To implement this intervention, CPS’ OSHW created a new Healthy CPS Network Specialist position to help schools implement these policies by providing TA and connecting schools to various resources (8). The Network Specialist used a Multi-Tiered System of Supports (MTSS) framework to provide schools with varying tiers of support in implementing Healthy CPS policies. Similar to the continuum of prevention model used to prevent substance use and to improve child welfare (9, 10), the MTSS framework is used by the education sector to provide “Tier 1” universal support to all schools/students, “Tier 2” targeted support to a subset of schools/students that require additional assistance, and “Tier 3” individualized support to schools/students with ongoing and intensive needs (11, 12). School-based interventions delivered using a MTSS framework have been shown to improve student mental health, social behaviors, school engagement, academic performance, and more (13, 14). The Network Specialist’s activities were aligned with Cook et al.’s SISTER strategies (3). One of the goals with the Network Specialist intervention was to not only test the efficacy of the Network Specialist position in helping schools improve Healthy CPS compliance but also, equally important for the district leadership, to provide data for CPS to determine the nature of the supports provided by the Specialist, the time spent on providing supports by MTSS Tier, and the cost to do so.

However, detailed time and cost data on intervention activities are often lacking in implementation science, even more so in evaluating school-based interventions (15, 16). Various methods have been developed to fill this gap. A common costing method is the top-down approach, which is one that allocates the overall time and cost spent at the organization level to the underlying implementation activities (17). This type of allocation method relies heavily on the richness and details of data on intervention activities to approximate or assume the distribution of time and cost (17). This approach often suffers from lack of accuracy, especially for evaluating interventions. Another costing method is the bottom-up approach, which aggregates individual-level data to derive the overall costs (17). This approach is often used in medical billing and fee-for-service applications, and is very time consuming and labor intensive (17). More modern and mixed approaches have been developed to overcome the challenges in these traditional methods. Saldana et al. developed a costing approach that mapped intervention costs to different stages of implementation (18). Based on a business accounting method (19), Cidav et al. also detailed a time-driven activity-based costing approach that mapped costs to various implementation strategies (20). By using these approaches, researchers can directly compare the time and costs of various intervention activities performed and the implementation strategies utilized. Previous studies have used activity-based time tracking (ABTT) tools in primary care and substance use interventions to assess the cost of intervention activities (21, 22). These studies highlight the utility of these approaches in implementation science, and the use of time tracking tools helps collect data on intervention activities that inform decision-makers about how to implement policies and interventions with fidelity.

In the methods section that follows, we describe the development of an ABTT tool for use by the Network Specialist in supporting schools with policy implementation. Then, in the results, we (1) document the utility and feasibility of using the tool; (2) illustrate how the ABTT tool was used to assess the time and cost spent on providing supports by mode of support, MTSS Tier, and SISTER domain; and (3) show how the data from the ABTT tool were used to demonstrate that the Network Specialist provided support for schools with the greatest need. We hypothesized that the Network Specialist would spend a greater proportion of time providing intensive Tier 3 supports, and that those supports would predominantly be targeted toward schools with the most need as measured by baseline Healthy CPS policy compliance. To the best of our knowledge, an ABTT tool has not been used before in school-based interventions to support schools in implementing health and wellness policies.

Development of the Tool

Use of a Community-Engaged Process for Developing and Improving the Tool

A community-engaged process (23) was taken to develop the ABTT tool, with iterative feedback from the Network Specialist and OSHW leadership to continuously improve the tracking tool for the end user (i.e., the Network Specialist) (24). Guided by existing literature on tracking implementation activities (18, 20), the initial version of the tracking tool used in school years (SY) 2020–2022 was developed in Google Sheets, CPS’ preferred platform, to capture the Network Specialist’s day-to-day activities because it could be easily integrated into CPS’ existing TA tracking infrastructure housed in Google Sheets. The Network Specialist recorded each interaction with schools on a weekly basis (as close to real time as possible), including basic information about the nature of the interaction (e.g., date, names and roles of persons interacted with, mode of contact, duration). An open-ended text field was also provided for the Network Specialist to describe the topic and content of the interaction.

However, upon reviewing the tracking tool data in 2022, it became apparent that it was not accurately capturing the details on the specific supports/strategies being used by the Specialist, nor could it be used to identify the MTSS Tiers of support provided by the Specialist. Thus, through a series of four meetings between the UIC research team and the Network Specialist, we developed a revised tool in Google Forms (for use in SY 2022–2023), similar to completing an electronic survey. The revised version of the tool was expanded to collect additional information on how TA support was delivered, including additional open-ended fields, close-ended questions on the most commonly provided support, and Healthy CPS topics discussed. This was done to standardize the tracking tool and to enhance its reliability. Standardization of tracking TA support across multiple end users was essential to the validity and sustainability of the tracking tool as CPS sought to expand its use to other CPS networks with the planned onboarding of additional Network Specialists. The use of the Google Form allowed for more streamlined data collection and reduced missing or inaccurate data through the use of forced responses and data validation. Figure 1 summarizes the main components of the tool presented in this study; Additional File 1 contains the specific questions and response options included in the tool for the items captured for this study. (A full-length version of the tool which captured specific Healthy CPS policy issues being addressed at each school is available upon request from the corresponding author.)

Insert Fig. 1 about here

Entries from both iterations of the ABTT tool were processed monthly by the UIC research team using Stata v.16 (StataCorp, College Station, TX, USA) to ensure data fidelity. Any inconsistencies or missing entries were reconciled with the Network Specialist via follow-up discussions and email communications. Summary statistics were also generated monthly and reviewed to monitor implementation fidelity and provide continuous quality improvement feedback to the Network Specialist on how to effectively use the tool. Between SY 2020–2023, a total of 78 interactions (total duration: 57.0 hours) took place between the Network Specialist and the research team for the purposes of continuous quality improvement of the tracking tool (e.g., via meetings/emails to clarify what interactions were being had). Face validity of the fields in the tracking tool was assessed through discussions with the Network Specialist to evaluate that their understanding of what each field was asking for aligned with what the research team had intended.

Determining the SISTER Domains and Tiers of Support used by the Network Specialist

Open-ended text fields, specifically questions 10 and 13 in the ABTT tool (see Additional File 1), were used by the Network Specialist to describe the topic and content of the interaction, resources shared, referrals made, and activities/strategies used to support schools’ health and wellness policy implementation efforts, as well as the tier of MTSS support provided. This open-ended format was used as Walsh-Bailey et al. found that an open-ended tool to record implementation strategies was more acceptable and feasible compared to a more structured tool forcing respondents to select from pre-populated options (25). Researchers also wanted the Network Specialist to be able to provide as much detail as possible, so that the interactions could be later reviewed and coded by the research team.

To code the open-ended text field data, entries were exported from the ABTT tool into an Excel spreadsheet, and columns were added to code the entries using SISTER implementation strategy domains and definitions put forth by Cook et al. (3). Using these initial code definitions for the implementation strategy domains, the open-ended field data were independently coded by three members of the research team (two lead investigators and one research associate). After this initial review of the entries and application of the SISTER domain codes, the research team held multiple meetings to further refine the SISTER domain code definitions to be more specific to the work of the Network Specialist and identified specific SISTER implementation strategies within each SISTER domain that the Network Specialist was using to help schools achieve Healthy CPS compliance (e.g., prepare and identify champions was identified as a strategy used by the Network Specialist and defined as preparing and identifying Wellness Champions to become leaders in implementing Healthy CPS in their schools). The research team then re-reviewed the initial application of SISTER domain codes and resolved any differences through discussion (and in consultation with the Network Specialist when additional details were needed about the nature of the logged interaction). After confirming agreement on the application of the SISTER domains, the research team then coded for the specific SISTER implementation strategies within each domain that mapped to the regular activities of the Network Specialist, and similarly met to discuss how the codes were being applied and rectify any disagreements in application. Investigators arrived at a final codebook that contained 6 SISTER domains and 15 SISTER strategies. For the sake of brevity and due to the limited number of total interactions coded, data in this study is presented at the implementation strategy domain-level (and not at the more specific implementation strategy-level).

In addition to applying SISTER domain and strategy codes, the ABTT tool data were analyzed by designating the MTSS Tier provided by the Network Specialist. The MTSS categories included Tiers 1, 2, and 3—with Tier 1 supports being the least involved supports that are provided to all schools and Tiers 2 and 3 representing more focused and the most targeted supports to schools, respectively, which were provided based on individual schools’ needs and Healthy CPS attainment status. When completing the log, the Network Specialist designated which tier of support they thought the interaction fell under – which the research team then reviewed when processing the data to ensure that the interaction was categorized correctly. The Network Specialist’s understanding of the differences in tiers was also periodically checked during process evaluation interviews throughout the study period. From the tool, 556 interactions were reviewed and only one interaction logged by the Network Specialist could not be categorized into any of the three tiers of support.

Determining the Utility and Feasibility of the Tool

Qualitative data were used to assess the utility and feasibility of using the ABTT tool. First, three key informant interviews between the Network Specialist and the UIC research team where the tool was discussed were held between 2021 and 2023. (Separate meetings were held with the Specialist to discuss tool development and refinement to ensure tool validity through continuous quality improvement of the data entry, but those meetings did not capture data on the use or feasibility of the tool.) Additionally, an asynchronous structured interview over email took place in May 2024 with the Network Specialist (one of the co-authors, CM) to gain insight into their perspective on the feasibility and usefulness of the tool (26).

Data Sources for Examining the Implementation Costs and Whether the Specialist Supported Schools with the Most Need

Cost Data

Consistent with previous studies evaluating school-based interventions, costs were measured from an education sector perspective and included only the programmatic costs (27, 28). This perspective is most relevant to decision-makers interested in implementing the intervention. Costs were grouped into three categories: personnel, travel, and equipment. Personnel costs were prorated based on the proportion of time the Network Specialist spent interacting with schools (13.7% in SY 2020–2021, 32.0% in SY 2021–2022, and 51.5% in SY 2022–2023), excluding their interactions with CPS, OSHW, and the research team that pertained to the design and planning of the intervention or the tracking tool. Travel and equipment costs were not prorated. Interactions between the Network Specialist and schools were relatively low in the first year due to the COVID-19 pandemic when Chicago schools were operating remotely and the Network Specialist was restricted in their capacity to interact with schools by the school district leadership. Personnel costs included the salary and benefits of the Network Specialist (1 full-time equivalent [FTE]) and the Healthy CPS Program Manager, who reported spending 10% of their time supporting the Network Specialist in implementing the intervention (0.1 FTE). Travel costs were derived from reimbursement forms submitted by the Network Specialist to CPS to cover their visits to schools to conduct on-site meetings with school leadership and provide TA. Mileage rates for travel were determined based on the standard mileage rate published by the Internal Revenue Service for business use (29). Equipment costs were self-reported and included the purchasing of CPS-issued electronic devices.

Healthy CPS Scores

To assess the extent to which the Network Specialist was providing targeted supports to schools with the greatest need at baseline, publicly available Healthy CPS annual data from SY 2018–2019 were compiled for each school through CPS’ School Profile Search (30). Healthy CPS for that SY assessed 32 policies pertaining to four health and wellness areas: 1) chronic disease management (e.g., whether staff completed the chronic conditions training), 2) health instruction (e.g., whether schools provided the required physical, nutrition, and sexual health education curriculum), 3) school health leadership and environments (e.g., whether the school has a Wellness Champion and team to facilitate wellness policy implementation, whether the school offered daily recess), and 4) health services (e.g., whether the school participated in dental and vision exam programs) (31). The Healthy CPS were collected through district administrative data and schools’ self-reported surveys. The SY 2018–2019 Healthy CPS score, computed as the averaged percentage score in complying with policies in each of the four health and wellness areas, serves as a baseline measure of a school’s need for TA before the Network Specialist intervention.

Analysis

Qualitative Analysis

To determine feasibility and utility of the tool, the qualitative interview data and the asynchronous email data were analyzed by two study authors (MO, EJR) using MaxQDA Pro (VERBI GmbH, Berlin, Germany). The transcripts and email were analyzed to identify specific feedback from the Network Specialist about the tool and how it was being used. The qualitative data were organized according to broad categories of feasibility and utility. Most of the interview data focused on feasibility, particularly strengths and weaknesses of using the tool, whereas the final email exchange captured specific information on the utility or value of using the tool.

Quantitative Analyses

ABTT tool data were pulled from Google Sheets or Google Forms monthly between June 2020 and June 2023. Any missing or inconsistent data were resolved through discussions with the Network Specialist. If an activity pertained to multiple schools, the duration was viewed as the overall time it took to complete the activity across the schools; thus, the duration was recoded by splitting the time equally among the schools involved. For example, if the Network Specialist spent 10 minutes to draft and send an email blast to all intervention schools, the overall duration of that interaction would be divided and attributed equally to each of the schools. Cost by mode of interaction, tier of support, and SISTER implementation domains were computed by the percentage of time spent by the Network Specialist.

For each tier of support, the schools served by the Specialist were categorized into tertiles (low, medium, and high) based on the total duration of interactions with the Network Specialist falling into the given tier. Mean SY 2018–2019 Healthy CPS scores were computed within each tier of support and tertile of interaction duration. A Wald test was performed to determine the statistical significance of differences in mean scores within each tier of support. All analyses were performed in Stata v.17 (StataCorp, College Station, TX, USA).

Between SY 2020–2021 and SY 2022–2023, the Network Specialist supported 33 unique schools with an average of 7,942 students per SY (Table 1). The students in the intervention (Network Specialist) network were predominantly Black and Hispanic, most students were eligible for free or reduced-price lunch, and about one-sixth of the students had limited English proficiency and required special education (32). This study was approved by the University of Illinois Chicago Institutional Review Board (protocol #2019 − 1161).

Table 1

School and student characteristics of the intervention network, school years 2020–2023.
	Count or Mean
Network
Number of unique schools	33
Average number of students	7,942
Average number of students per school	310
Student race/ethnicity (%)
American Indian/Alaska Native, non-Hispanic	0.1
Asian, non-Hispanic	0.2
Black, non-Hispanic	63.5
Hispanic	33.4
Native Hawaiian/Pacific Islander, non-Hispanic	0.0
White, non-Hispanic	2.0
Multiple race, non-Hispanic	0.6
No race/ethnicity available	0.0
Other student characteristics (%)
Limited English proficiency	16.2
Special education	16.2
Free/reduced-price lunch eligible students	83.3
Statistics were computed for each school year then averaged across the three school years.

Feasibility and Utility of the Tool

Early feedback from the Network Specialist when using the Google Sheets version of the tool highlighted that given the extensive amount of interactions that the Specialist was having across the schools, the tool was critical in helping them keep track of the “wealth of information” that was being exchanged between the Specialist and each school. In addition, the Specialist noted that the log was critical in helping with annually revising the Network Specialist Implementation Guide, which is used to ensure fidelity of the Network Specialist intervention (8).

However, prior to the conversion to the Google Form, the Specialist noted that “…It can get kind of tricky…that communications log can get kind of tricky for me, because I do interact a lot, … it can be tough just to make sure that I'm thinking about as many interactions as I have and trying to log them.” The early interviews also helped to identify that the Specialist was entering interactions with Network and OSHW staff that had to do with specific schools on separate Google Sheets related to the Specialist’s interaction with the Network or OSHW because someone from the Network or OSHW was also involved. That helped us realize that a lot of the interactions we had heard about from the Network Chief or OSHW staff were missing in our initial analyses of the log data using only the school tab in Google Sheets. We then went back and reviewed the entries from the other tabs to ensure that we did not miss any school-level interactions for our analyses.

After the first full year of using the Google Form, the Network Specialist expressed that the tracking tool was feasible and useful, saying “the log is extensive, but doable. [It’s] something I have to carve out time to complete, but the data collection value makes it worth the effort and time in my opinion.” Completing the tracking tool also allowed the Network Specialist to provide more tailored support to schools, stating “it keeps me on track with what I'm working on and if I've been on top of my TA for all of my schools. It puts the TA work at the forefront of my mind for each school I'm interacting with.” The Specialist further noted that the log helps her keep a “record of the work [she’s] been doing with each school and hold myself accountable to continue TA with said schools.” And it enabled her to better keep track of the TA that she was (or was not) providing: “The TA log helps me to be intentional with my time and reflect on what TA I have or haven't been providing. If I see that I haven't logged a TA entry for a certain school or area in a while it alerts me that I need to make a move and get that TA up.”

Importantly, CPS plans to continue using the tool for use by future Network Specialists. As noted by the Network Specialist, “Now that I've transitioned into a Team Lead position I want to use the TA log to monitor the work that the specialists I'm supervising are doing. It's a great way to collect data and monitor the progress of our specialists and know what work is being done in their network. I think this TA log also helps keep whoever is filling it out accountable and keep them on track to be providing the best TA possible.”

Network Specialist Interactions and Costs

The Network Specialist spent a total of 256.1 hours over the three SYs interacting with schools (SY 2020–2021: 41.6 hours; SY 2021–2022: 99.3 hours; SY 2022–2023: 115.2 hours), predominantly through emails, videochats, and in-person meetings (Table 2). As previously mentioned, interactions between the Network Specialist and schools were restricted by school leadership in the first year of the study due to COVID-19 (virtually all of the interactions in this year were by email). After prorating the costs for the Network Specialist’s interactions with schools, the intervention cost a total of $94,486 over the study period. The overwhelming majority (99.6%) of the intervention costs were derived from personnel expenses. This is in line with our intervention design that centered around the Network Specialist providing TA to schools. Only 0.2% ($224) of the costs derive from transportation, in part due to there being no on-site visits to schools during the first year of the intervention. Another 0.2% ($198) of the costs derive from equipment purchased.

Table 2

Percentage of Network Specialist time spent and cost supporting schools by mode of interaction, tier of support provided, and SISTER implementation domains utilized.
	Duration of interactions, %	Cost, $
Mode of interaction
Email	39.8	37,605
Videochat	34.4	32,503
In-person	21.8	20,598
Phone	3.6	3,401
Not available	0.4	378
Tier of support
Tier 1	26.7	25,228
Tier 2	31.7	29,952
Tier 3	41.4	39,117
Not available	0.2	189
SISTER implementation domains (multiple may apply)
Use evaluative and iterative strategies	38.6	36,472
Train and educate stakeholders	36.8	34,771
Support educators	34.6	32,692
Develop stakeholder interrelationships	28.0	26,456
Provide interactive assistance	7.1	6,709
Engage consumers	2.0	1,890

Just over one-quarter (26.7%) of the Specialist’s time and cost ($25,228) was spent providing universal Tier 1 support (mean duration per interaction: 3.7 min [not shown in tables]), which included sending general information and resources to all schools. About one-third (31.7%) of the Specialist’s time and cost ($29,952) was spent providing more targeted Tier 2 support (mean: 14.5 min), which included responding to specific school requests for resources and connecting schools with specific services. The Network Specialist spent the highest proportion of time, 41.4%, and cost ($39,117) providing intensive Tier 3 support (mean: 40.3 min), which included regular check-in meetings with school leadership to discuss Healthy CPS achievement progress, set goals, and create plans of action.

To support schools in implementing Healthy CPS policies, the Network Specialist used strategies that fell under the following SISTER domains: use evaluative and iterative strategies (38.6% of time spent, cost: $36,472); train and educate stakeholders (36.8%, cost: $34,771); support educators (34.6%, cost: $32,692); develop stakeholder interrelationships (28.0%, cost: $26,456); provide interactive assistance (7.1%, cost: $6,709); and engage consumers (2.0%, cost: $1,890). Three SISTER domains (adapt and tailor to context, use financial strategies, and change infrastructure) were not applicable to this study.

Insert Table 2 about here

As hypothesized, the Network Specialist’s tailoring of supports to schools was also reflected by the tracking tool. Schools with the highest duration of Tier 3 interactions with the Network Specialist had significantly lower average baseline SY2018-2019 Healthy CPS scores (score: 64.4) compared to schools with medium interactions (score: 67.3) and schools with low interactions (score: 78.4; p = 0.045; Table 3). In line with our hypothesis, our findings show that the Network Specialist targeted the most intensive Tier 3 supports to schools that most needed the support. There were no differences in baseline Healthy CPS scores by the duration of Tier 1 and Tier 2 supports provided.

Table 3

Baseline Healthy CPS mean scores by duration of interactions for each tier of support.
Baseline SY2018-2019 Healthy CPS score, mean	Low duration interactions	Medium duration interactions	High duration interactions	p-value
Tier 1 support	77.1	64.7	69.6	0.090
Tier 2 support	69.7	77.4	64.3	0.092
Tier 3 support	78.4	67.3	64.4	0.045
Each cell represents the mean Healthy CPS baseline score (maximum score: 100) given the Tier of support and duration of interaction category. Interaction duration for Tier 1 support (low: 15–125 min; medium: 130-159.3 min; high: 159.5–207 min; N = 11 schools for each tertile), Tier 2 support (low: 5–90 min; medium: 93–184 min; high: 200–363 min; N = 11 for each tertile), and Tier 3 support (low: 0-135 min, N = 12; medium: 136–226 min, N = 11; high: 241–446 min, N = 10). P-values were computed using a Wald test of means within each tier of support.

Implementation science literature has long called for more detailed tracking of time and cost data associated with implementation activities (15, 16). To our knowledge, this study is the first to use an ABTT tool in a school-based intervention to support health and wellness policy implementation. The tracking tool employed in this study was instrumental in describing the types and intensity of support the Network Specialist provided and the implementation strategies used. Without the use of the tracking tool, it would not have been possible to determine the duration and tiers of support provided to each school or to compute the proportion of the Network Specialist’s time spent on implementing the intervention compared to its design and planning. Our findings on the types of support provided and the implementation strategies used are useful for future researchers and policymakers in replicating and implementing this intervention with fidelity, in combination with the implementation guides produced as part of the larger intervention evaluation (8).

In the study by Ritchie et al. which applied a similar tracking tool to assess the time and cost of facilitators implementing primary care mental health integrations, they found that different facilitators performed distinct implementation activities (21). Through the tracking tool, our study was able to show that the Network Specialist provided tailored supports to different schools and that the most intensive Tier 3 supports were targeted toward schools with the most need. This was achieved through the Network Specialist building close interrelationships with school leadership and providing targeted and specific TA according to each school’s needs.

Though there are burdens and challenges to implementing an activity-based time tracking tool, it has been viewed as feasible to accomplish (33). However, intervention evaluations often do not elicit stakeholder perspectives on these tools in practice. In our study, the Network Specialist recounted the value of the tracking tool in providing continued and tailored supports to schools, particularly as the intervention is expanded to more schools. This highlights the importance of developing the tracking tool in close collaboration with the end user to maximize its usefulness. Through the community-engaged approach to develop the tracking tool and continuous feedback and improvements made to the tracking tool, we yielded a final product that is feasible to implement, valuable to both CPS and the end user, and sustainable as the tracking tool remains in use by CPS even outside of the evaluation period. As CPS expands support to additional schools and networks, the tracking tool can allow close monitoring of the TA provided by different Network Specialists.

Limitations

This study should be considered in light of the following limitations. First, the activity-based time tracking tool data were self-reported by the Network Specialist. However, the research team conducted continuous quality assurance of the data and discussed data and reporting standards with the Network Specialist on a regular basis to ensure that the tracking tool was used consistently and reliably. Second, the Network Specialist was restricted from contacting schools or providing in-person TA in the first year of the study due to COVID-19. This reduced the overall duration of interaction between the Network Specialist and schools and undercounts the intensive Tier 3 supports that would have otherwise been provided in year one. Third, there was a hiatus in the TA provided to schools between June to October of 2022 as the Network Specialist in the first two years of the study transitioned out of that role and a new Network Specialist was hired for year three of the study. This led to an undercounting of support that would have been provided.

In a school-based intervention to improve health and wellness policy compliance, an activity-based time tracking tool was used to measure the types and intensity of support provided to schools. In accordance with the intervention design, the tracking tool showed that the most intensive supports were targeted toward schools with the most need based on baseline policy compliance. The tracking tool was both feasible and valuable to the end user in providing tailored support. Our findings emphasize the need for utilizing similar tracking tools in school-based implementation science studies and in intervention evaluations.

ABTT	Activity-based time tracking
CPS	Chicago Public Schools
FTE	Full-time equivalent
MTSS	Multi-Tiered System of Supports
OSHW	Office of Student Health and Wellness
SISTER	School Implementation Strategies, Translating ERIC Resources
SY	School Year
TA	Technical assistance

Ethics approval and consent to participate

This study was approved by the University of Illinois Chicago Institutional Review Board (protocol #2019 − 1161) and the Chicago Public Schools Research Review Board (protocol #1582).

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Authors Information (Optional)

Yu Chen Lin drafted and conducted this study while employed at the University of Illinois Chicago.

Funding

This study was supported by the Health Promotion and Disease Prevention Research Center cooperative agreement, Centers for Disease Control and Prevention, U.S. Department of Health and Human Services (HHS) as part of a financial assistance award totaling $750,000. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by CDC/HHS, the U.S. Government, or the authors’ employers.

Authors' contributions

JFC conceptualized the study and obtained funding. YCL designed the data collection instruments and analysis plan. AW, CM, EJR, and MO collected the data. YCL and MO analyzed the data. YCL, MO, and JFC drafted the manuscript. JL and EJR provided feedback and substantive revisions. All authors read and approved the final manuscript.

Acknowledgements

Not applicable.

Availability of data and materials

The datasets generated and analyzed during the study are not publicly available due to the sensitive nature of the time and cost data but are available from the corresponding author on reasonable request. However, the instrument we used for time tracking is available upon request from the corresponding author.

Chriqui JF, Leider J, Temkin D, Piekarz-Porter E, Schermbeck RM, Stuart‐Cassel V. State laws matter when it comes to district policymaking relative to the whole school, whole community, whole child framework. J Sch Health. 2020;90(12):907–17.
Chriqui JF, Leider J, Turner L, Piekarz-Porter E, Schwartz MB. State wellness policy requirement laws matter for district wellness policy comprehensiveness and wellness policy implementation in the United States. Nutrients. 2021;13(1):188.
Cook CR, Lyon AR, Locke J, Waltz T, Powell BJ. Adapting a Compilation of Implementation Strategies to Advance School-Based Implementation Research and Practice. Prev Sci. 2019;20(6):914–35.
Chicago Public Schools. Healthy CPS n.d. [ https://www.cps.edu/strategic-initiatives/healthy-cps/
Chicago Public Schools. Stats and Facts n.d. [ https://www.cps.edu/about/stats-facts/
Chicago Public Schools. Networks [ https://www.cps.edu/schools/networks/
Chicago Public Schools. School Data Chicago, IL: Chicago Public Schools; n.d. [ http://www.cps.edu/SCHOOLDATA/Pages/SchoolData.aspx
Elizabeth Jarpe-Ratner MO, Sabrina Arancibia JF, Chriqui. Cassidy Malner, Kathryn Ramirez-Mercado, Jamie Tully, Tarrah DeClemente. Healthy CPS Network Specialist Implementation Guide [Internet]. Policy, Practice and Prevention Research Center, University of Illinois at Chicago, and Chicago Public Schools; 2023 [ https://indigo.uic.edu/articles/online_resource/Healthy_CPS_Network_Specialist_Implementation_Guide/25742859
Substance Abuse and Mental Health Services Administration. Focus on Prevention 2017 [ https://store.samhsa.gov/sites/default/files/sma10-4120.pdf
Capacity Building Center for States. Working across the prevention continuum to strengthen families 2021 [ https://capacity.childwelfare.gov/sites/default/files/media_pdf/prevention-continuum-strengthen-families-cp-20119.pdf
Sugai G, Horner RH. Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality. 2009;17(4):223–37.
Goodman S, Bohanon H. A framework for supporting all students: One-size-fits-all no longer works in schools. Am School Board J. 2018.
Nitz J, Brack F, Hertel S, Krull J, Stephan H, Hennemann T et al. Multi-tiered systems of support with focus on behavioral modification in elementary schools: A systematic review. Heliyon. 2023.
Bohnenkamp JH, Hartley SN, Splett JW, Halliday C, Collins D, Hoover S, et al. Promoting school safety through multi-tiered systems of support for student mental health. Preventing School Failure: Altern Educ Child Youth. 2023;67(1):9–17.
Lau R, Stevenson F, Ong BN, Dziedzic K, Treweek S, Eldridge S, et al. Achieving change in primary care—effectiveness of strategies for improving implementation of complex interventions: systematic review of reviews. BMJ open. 2015;5(12):e009993.
Wolfenden L, McCrabb S, Barnes C, O'Brien KM, Ng KW, Nathan NK et al. Strategies for enhancing the implementation of school-based policies or practices targeting diet, physical activity, obesity, tobacco or alcohol use. Cochrane Database Syst Reviews. 2022(8).
Mogyorosy Z, Smith P. The main methodological issues in costing health care services: a literature review. Centre for Health Economics, University of York Working Papers. 2005(007cherp).
Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The cost of implementing new strategies (COINS): a method for mapping implementation resources using the stages of implementation completion. Child Youth Serv Rev. 2014;39:177–82.
Kaplan RS, Anderson SR. Time-driven activity-based costing: Børsen; 2008.
Cidav Z, Mandell D, Pyne J, Beidas R, Curran G, Marcus S. A pragmatic method for costing implementation strategies using time-driven activity-based costing. Implement Sci. 2020;15:1–15.
Ritchie MJ, Kirchner JE, Townsend JC, Pitcock JA, Dollar KM, Liu C-F. Time and organizational cost for facilitating implementation of primary care mental health integration. J Gen Intern Med. 2020;35:1001–10.
Garcia CC, Bounthavong M, Gordon AJ, Gustavson AM, Kenny ME, Miller W, et al. Costs of implementing a multi-site facilitation intervention to increase access to medication treatment for opioid use disorder. Implement Sci Commun. 2023;4(1):91.
Clinical and Translational Science Awards Consortium. Principles of community engagement, 2nd ed. Bethesda, MD: US Department of Health and Human Services, National Institutes of Health. 2011. https://www.atsdr.cdc.gov/communityengagement/pdf/PCE_Report_508_FINAL.pdf
Key KD, Furr-Holden D, Lewis EY, Cunningham R, Zimmerman MA, Johnson-Lawrence V, et al. The continuum of community engagement in research: a roadmap for understanding and assessing progress. Progress community health partnerships: Res Educ action. 2019;13(4):427–34.
Walsh-Bailey C, Palazzo LG, Jones SM, Mettert KD, Powell BJ, Wiltsey Stirman S, et al. A pilot study comparing tools for tracking implementation strategies and treatment adaptations. Implement Res Pract. 2021;2:26334895211016028.
Spencer T, Rademaker L, Williams P, Loubier C. Online, asynchronous data collection in qualitative research. Popularizing Sch Research: Res Methods Practices. 2021:327.
Finster MP, Feldman J. Cost-Effectiveness of 2 Support Models for a Healthy School Initiative. J Sch Health. 2020;90(9):724–30.
Lane C, Nathan N, Reeves P, Sutherland R, Wolfenden L, Shoesmith A, et al. Economic evaluation of a multi-strategy intervention that improves school-based physical activity policy implementation. Implement Sci. 2022;17(1):40.
Internal Revenue Service. Standard mileage rates [Internet]. [ https://www.irs.gov/tax-professionals/standard-mileage-rates
Chicago Public Schools. School Profile Search [Internet]. [ https://www.cps.edu/schools/find-a-school/
Chicago Public Schools. School Year 2018–2019 Healthy CPS Alignment Report District Report 2020 [ https://drive.google.com/file/d/1y1Kkdb0zINbeJXK5_zhVt1_ilKeRCvN_/view
Chicago Public Schools. Demographics [Internet]. [ https://www.cps.edu/about/district-data/demographics/
Levy DE, Singh D, Aschbrenner KA, Davies ME, Pelton-Cairns L, Kruse GR. Challenges and recommendations for measuring time devoted to implementation and intervention activities in health equity-focused, resource-constrained settings: a qualitative analysis. Implement Sci Commun. 2023;4(1):108.

Download PDF

Reviewers agreed at journal
12 Aug, 2024
Reviewers invited by journal
12 Aug, 2024
Editor assigned by journal
07 Aug, 2024
First submitted to journal
06 Aug, 2024

You are reading this latest preprint version

Use of an activity-based time tracking tool to support implementation of a school district-level technical assistance intervention

Status:

Version 1

Abstract

Figures

Contributions to the literature

BACKGROUND

METHODS

Development of the Tool

Use of a Community-Engaged Process for Developing and Improving the Tool

Insert Fig. 1 about here

Determining the SISTER Domains and Tiers of Support used by the Network Specialist

Determining the Utility and Feasibility of the Tool

Cost Data

Healthy CPS Scores

Analysis

Qualitative Analysis

Quantitative Analyses

RESULTS

Feasibility and Utility of the Tool

Network Specialist Interactions and Costs

Insert Table 2 about here

DISCUSSION

Conclusions

Abbreviations

Declarations

Competing interests

Authors Information (Optional)

Funding

Authors' contributions

Acknowledgements

Availability of data and materials

References

Supplementary Files

Status:

Version 1