Early Warning Systems Achieve Their Goals When Data is Tied to an Intervention

doi:10.21203/rs.3.rs-3617115/v1

Download PDF

Case Report

Early Warning Systems Achieve Their Goals When Data is Tied to an Intervention

https://doi.org/10.21203/rs.3.rs-3617115/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Early Warning Systems (EWS) are research-based analytics that use statistical models to assess dropout risk. School leaders use this analytic to consolidate data about a student and provide actionable data to craft an intervention. Little is currently known about the processes involved in school implementation. This is important as it guides use of the tool, especially for those schools using data from Statewide Longitudinal Data Systems. Our study analyzes the Montana EWS data for the past ten years to see if data is used in interventions whose outcomes demonstrate a linkage back to the data. This study distinguishes between schools - high, low, and non-adopters based on an opt-in model. High adopters implement the model based on shared vision, value, and communication practices. There are questions whether low adopters engaged the data and if they offer their students dropout prevention services. Non adopters have many core traits (student outcome and institutional) indicating that there is less of a demand in these school communities. The scale of the program is determined by whether schools opt-in and set priorities for dropout prevention. High adopters share many indicators of disadvantage as low adopters. The difference is that certain school level factors (e.g., long teacher tenure) enabled high adopters to prioritize dropout prevention and integrate data in the process.

For the last two decades, Early Warning Systems (EWS) have been used in dropout prevention. These systems provide objective indicators to frame risk based on analytics which have been trained against prior year(s) data. They are often used in conjunction with other inputs, including feedback from parents and teachers. The degree of communication of EWS data and the feedback loop to improve it highlight traits of a successful intervention. Often this is done in the context of Multi-Tiered System of Supports (MTSS) interventions which allocate interventions according to student need, the local capacity to support an intervention, and the priorities to make interventions happen in each context.

The Montana EWS calculates the dropout probability for current students and provides an assessment of risk factors (academic, attendance, behavior, and mobility). These models often have a strong association between the dropout probability based on certain risk factors and any eventual drop out status. Models involving successful dropout prevention practices would have less of a direct association between dropout probabilities and drop out status since many students found at one time to be at risk could have participated in evidence-based intervention programs and graduated instead. This opens the door to inquiries about the effectiveness of the system in Montana. Research on data use of EWS systems is sparse, and little was previously known about the case of Montana or Statewide Longitudinal Data System (SLDS) early warning systems (Faria, Sorensen, Heppen., Bowdon, Taylor, Eisner, & Foster, 2017; Marken, Scala, Husby-Slater & Davis, 2020).

EWS research should focus on intervention and the systemic conditions which enable successful interventions. Data use of an EWS can be much richer than early identification. Educators may know who their at-risk students are, but they may not know how to operationalize the data to make a successful intervention over time. The actual linkage to bring this data to inform interventions, and change these interventions over time, is key to program effectiveness. This involves framing the need for EWS interventions, the vision and value found in the data, the communication strategies involved, and a focus on the priorities that schools may have to implement EWS inspired interventions. The process of intervening is the missing piece to the puzzle of what makes an EWS effective: how it is occurring in EWS schools, in what form, and to what intensity. There are indications that research on processes involved in EWS nationally is sparse and remains so (Bruch, Gellar, Cattrell, Hotchkiss & Killewald, 2020; Faria, et al., 2017; Marken, et al., 2020). In literature and practice often, there is a narrow focus on early identification with little to no follow up.

By investigating the data culture surrounding use of the tool and the pivotal role of interventions in the process, our study focuses on an established model in Montana that has been serving Montana’s students and schools for over 10 years. By investigating trends in its development, especially in the years before the pandemic, we are better able to understand how participating schools have implemented data driven dropout prevention strategies and if there are signs of a renewed focus following the pandemic on dropout prevention among school classified as ‘high adopting.’

We investigate the process of tying data to an intervention and how this process may be impacted by policy choices. An example of a policy change is when a school leader decides to focus on attendance and the school offers incentives to students to celebrate milestones in student growth. The measure is the degree to which evidence-based change is occurring when targeted interventions are based in policy change at the school level. Our research questions include:

What policy changes occur in schools that have used EWS in their intervention model?
Has EWS data inspired targeted interventions among participating schools?

Certain themes are apparent in these questions. Most importantly the impact of the development of a data culture, relationship building, early identification, longitudinal analysis, evidence-based interventions, and progress monitoring.

It is possible to identify students that are in danger of dropping out using established individual and school level indicators that can give warning to the signs of potential dropout (Bruch et al., 2020; Frazelle & Nagel, 2015; Koon & Petcher, 2015; O’Cummings & Therriault, 2015; Marken, et al., 2020). By focusing on attendance, behavior, mobility, and academics, the Montana EWS identifies intervention thresholds in a manner that is independent of demographics, indicators of economic disadvantage, or factors that may be out of a student’s control (e.g., foster care or disability).

In the Montana EWS, the scope and processes of an intervention are determined at the school level. They reflect policy choices. Another example of a policy choice is a school’s decision to address academics through a MTSS process; something which often had been linked to behavior but not to attendance and academics. As a result, either a narrow or broad application of the EWS tool is used by school leaders. Policy choices guide interventions. Early identification does not cause an intervention. A step is taken by the school to take that data and frame it into strategies for dropout prevention over time. This addresses vision, value, dissemination of EWS data, and communication. Interventions and follow up reveal the intensity of the data culture surrounding dropout prevention. Indeed, progress monitoring shows how embedded dropout prevention strategies are in the school.

Montana Office of Public Instruction (OPI) outreach involved developing and refining the diagnostic tool and engaging in professional development on best practices to use it. In this context outreach focused on navigating the online platform, managing, and uploading data, interpreting the results, and bringing these results to a MTSS committee. The EWS program pursued these four overarching goals:

1. Create and maintain a statistical model that accurately predicts the odds of a student dropping out (development framework).

2. Identify at-risk students before they drop out (professional development).

3. Help schools identify factors that are impacting each student’s dropout risk to prioritize and target interventions (professional development).

4. Help schools understand dropout risk trends at the school level to make decisions regarding policy and programs that may influence dropout risk (professional development).

The diagnostic tool creates profiles by school and student. EWS data allows for systematic planning by school teams. This planning builds off this ability to identify students and engage in the more difficult task of assigning students to appropriate and effective interventions. The principal element of the diagnostic tool is the interface that users see with the results of the logistic regression modeling by student and grade level. Once the system was in place and business rules governed the refinement and training of the data each year, the onus of Montana EWS was on professional development. This professional development was supported by data stewards at the OPI and individuals who worked in participating schools, such as a dedicated OPI data coach.

OPI provides professional development about the use of the data and provides insights into what triggering events will start, revise, and end an intervention. The question remains as to the degree to which interventions are data informed. Variation in implementation informs the measurement of the level of adoption. In many cases, high adopters are most closely aligned with the requirements of an effective implementation. In these schools, users take the data and communicate it to all members of their dropout prevention team (formal). In some high adoption schools, administrators and faculty demonstrate semi formal practices limiting dissemination to a small number of team members using the data. In low adoption schools, schools which expressed interest in the model but did not implement it, a nonformal approach to using the tool verified school data. Generally, these schools did not act on the Montana EWS data. This addresses school level policy priorities evidenced by the local demand for interventions and the EWS innovation.

To complete our investigation into effectiveness, we collected data regarding the degree of implementation of the EWS model using SLDS data, interviews, and surveys. SLDS analysis focused on contextual, institutional, and student outcome variables found in the Montana SLDS. The focus was on identifying the profiles of demand for the program by level of adoption. We used data from all schools in Montana to analyze three categories of schools: high adopters, low adopters, and non-adopters. The schools differ based on interest in the model, the demand to address dropout prevention, the vision to implement and use the tool, the values placed on identifying and acting on the needs of at-risk youth, and the dissemination practices in which all actors, including counselors and teachers, have access to the EWS data. The point of first upload is where schools established an EWS profile, interest was shown, and patterns of adoption may be assessed. This allowed us to distinguish between low and high adopters. We categorized schools by adoption status based on upload count, dissemination patterns, and factors in the formation of a data culture (e.g., vision and values) as found in the interview and survey data. Data was analyzed using a cross tab feature with Chi Square analyses (categorical data), a general linear model for comparison between groups with a continuous dependent variable, and a linear regression model for comparison between continuous dependent and independent variables.

Interview data was collected in November 2022. The focus of the interviews was on the level of adoption and questions about processes and policies at the school level. 15 schools were purposely sampled from a pool of 20 schools that expressed interest. Schools were selected based on socio-economic criteria, school size, and schools with high proportions of native students. We coded interview transcripts via an inductive process. The responses were open coded that allowed certain patterns to emerge. Open coding is the process of systematically interrogating the data by asking questions relevant to emerging themes, verifying these themes across interviews, and organizing the emerging analytical framework based on the findings. Themes were constructed that emerged from the coding framework and a deductive last stage integrates survey and SLDS data.

The survey addresses the intensity of EWS implementations including providing data on the scope of interventions. It seeks data about how schools use the EWS, interest in the EWS, and how schools transition from data to intervention. Survey data was used to confirm the findings of the SLDS data (school demand) and interview data (vision, value, dissemination, and communication). The sampling process was purposeful between groups (types of districts/schools) and random within groups (who in the district responded to the survey). Chi square analyses (categorical data) focused on years of implementation (six or more / five or less years), size of the district (less than 450 / greater than 450 students), number of individuals using the EWS data by district (5 or more / 4 or less). Invitations for the interview and survey were sent to all active adopters including superintendents, building leaders, counselors, and all staff that have access to their district’s SLDS portal.

What policy changes occur in schools that have used EWS in their intervention model?

Leadership of data driven dropout prevention models is determined by the requirements of the innovation, the identified demand in the school community, the vision to implement the model, the value found in the innovation, and the communication of these factors to the school community. Much of the early research behind EWS and the relationship to dropout prevention programs occurred in the context of the No Child Left Behind legislation and expectations placed on schools and accountability systems surrounding graduation. As a result, Early Warning Systems were paired with dropout prevention programs and behavioral strategies as seen in the Response to Intervention framework (RTI). In Montana, use of RTI models were sponsored by the OPI with extensive outreach by the agency to foster tiered supports in classrooms and schools. Their adoption represented a major policy change which has been evolving with MTSS.

Policy choices are critical to the operation of MTSS teams at the school level. MTSS frameworks arose out of the need to implement evidence-based strategies at scale in the context of over identification of students for services, as seen acutely with special education students. Benefits of the program are that students could have access to data driven supports based on identified need as seen in early warning systems or other forms of universal screenings. Moreover, the system allows interventions to be monitored over time for relevance and progress. This monitoring is an indicator of high adoption schools which were most likely to use EWS tools to inspire intervention.

The MTSS model in many Montana schools brought together attendance, academic, and behavioral supports under the same framework, something which had been separated under the RTI framework. The classification of interventions focuses not on the student’s behavior or circumstances, but rather on the level of support a student may receive. Tier 1 interventions are given to all students based on identified need– Universal. Tier 2 inventions are moderately intensive and often focus on small groups of students in the context of academic supports. Tier 3 is more intensive and focuses on the individual supports a student may need to be successful (Marken et al., 2020). One example of a Tier 3 intervention is Check and Connect; a faculty led mentoring program.

Interview respondents from high adoption schools remarked that the intimacy of their schools allowed for close relationship by multiple actors, dissemination of data to these actors, and allowed interventions to focus on student and family needs. These schools were generally from challenged communities but had an administration and faculty that closely sponsored dropout prevention and used it in all contexts in which a student demonstrated need. Despite being challenged, these communities had rich administrative and faculty resources as evidenced by low pupil-teacher ratios, longer faculty tenure in their position, and relatively high teacher and administrator salaries in comparison to other disadvantaged schools without these resources.

Demand

A sign of how data may be used is based on the demand for the EWS innovation in Montana schools. There is a similar profile among all kinds of adopters in comparison to non-adopters. A school’s demand for the EWS can be seen in various indicators, including outcome and institutional. An understanding of this demand informs state policy makers about the likelihood of scaling up the EWS program. The composition of the school community based on locale is significant (p = 0.000). In high adoption schools, equal percentages of schools are in town (45.56%) and rural areas (45.56%). This compares to the town (21.71%) and rural (63.57%) from schools that are low adoption. What we see in the rates of low adoption versus high adoption is that low adoption occurs more frequently in rural areas.

A measure of economic disadvantage is based on data from the American Community Survey. The Spatially Interpolated Demographic Estimate (SIDE) for these schools was significant (p = .002). In high adoption schools (247.96), there is significantly more economic disadvantage than in low adoption schools (257.50) and with non-adopters (267.60). The line between high adoption schools and low adoption schools is significant with high adoption schools experiencing more economic disadvantage. Similar trends can also be seen based on demographics. The percentage of minority students is significantly higher in high adoption schools and low adoption schools in comparison to non-adopters.

A variety of outcome data can be found with these schools. First, cohort graduation rates were higher (93.21%) among non-adopters in comparison to 86.50% among low adoption schools and 86.24% for high adoption schools (p = .001). Many rural high schools were non adopters. Rural high schools tend to have higher graduation rates than town or city high schools. Schools in the last category (non-adopters) may not have a defined demand for dropout prevention policies. The satisfactory attendance rate is the count of students who have 95% average daily attendance rate by the total number of students enrolled. Satisfactory attendance (p < .001) rates are also higher among non-adopters (49.24%) in comparison to low adoption schools (40.39%) and high adopters (40.16%). This highlights the degree of the challenges in these schools. Schools with lower satisfactory attendance rates would have a higher proportion of students that have issues surrounding daily attendance. This demand may have driven interest in the tool and the ability to find value in the tool.

Variation is also noted in assessment scores. Students in non-adopting schools are more proficient (50.71%) on the ELA assessment in grades 3–8 than their peers in high adoption schools (44.43%) and low adoption schools (42.88%) (p = 0.002). The trends regarding the ACT composite average (the 11th grade statewide assessment) are also significant (p = 0.020) and show that the non-adoption group scores higher (19.54) than the low adoption schools (18.54) and high adopters (18.72). Across all indicators, non-adoption schools stand out as having less of a defined demand in their schools for an EWS. This addresses how the school prioritized dropout prevention and the use of the EWS tool. It also raises issues with the scope and scale of the EWS program. Evidence suggests that the scale of the EWS program has eclipsed, signaling that higher levels of demand are shown by schools that have already shown interest in the tool. This impacts the process of scaling up the EWS insofar as renewed outreach could be targeted to adopters versus non adopters.

School Level Factors

Schools identify the demand for the innovation in tandem with developing the vision to implement the innovation and the process of finding value in the use of the tool. Among high adoption schools, the vision and value of the tool originates from the school leader in the context of an intervention team. This intervention team is one marker of the intensity of how well this vision and value is shared. It is exactly the process of meeting identified needs, finding value, and frequently returning to the data that demonstrates the intensity of data use that distinguished between different kinds of adopters for schools in all locales. Among high adoption schools this involved formal dissemination of EWS data to all stakeholders of the school’s intervention team.

One mediating factor to EWS dissemination is the degree a school has instructional and non-instructional staff participating in the innovation. To explore the hypothesis that the presence of faculty and certain non-instructional staff is related to the incidence in which a school has shared EWS data with the Montana Office of Public Instruction (OPI), we explored five different regression models to see how much, for example, is the incidence of upload explained by the pupil teacher ratio. Upload is important since schools must provide data to generate results of the Montana EWS. It’s one marker of the intensity of use of the EWS data. We found that pupil teachers ratios have a moderate association with upload frequency (r² = 0.230). The higher the per pupil ratio the more likely a school would upload more frequently. This differs from what we expected – lower pupil teacher ratios could yield more intense teacher focus on dropout prevention. We also looked at counselor ratios, psychologist ratios, social worker ratios, and librarian ratios and found that they explained little of the variation in average uploads / year. This indicates that staffing did not drive upload, however there was a reliance on other factors such as vision, value, and communication.

Use of a EWS encourages engagement with established school level procedures for support. Regardless of the level of adoption of the EWS program, users engaged MTSS in the same manner although at different levels and degrees of intensity. Consistency of intervention is important. One district remarked when switching over to a vendor EWS they had the same process (identifying data to inform interventions) as they used with the Montana EWS. Of the fifteen schools interviewed, eleven identified that they had a MTSS system in place, were in the process of constructing formal procedures, or they were in the process of reviving MTSS procedures that had been suspended because of the pandemic.

Just as important as the frequency of identification in EWS schools is the tie made between the intervention and the data. In schools with more than six years of experience with the EWS there is a clear linkage between the diagnostic tool and the intervention, where users more frequently claim this link occurs most of the time (p = .031). This may indicate that schools with longer exposure to the EWS tool focused on the development of this data culture surrounding dropout prevention which reportedly takes time and vision. They were also more likely to use the MTSS committee to address attendance and academic matters involved in dropout prevention.

A principal summarizes the kinds of interventions in their school (which are common in other schools that were interviewed): “We have some one-on-one support in the classroom with a push-in model with paraprofessionals. We have pull out support in a traditional fashion in which kids are pulled out for support one on one and in small groups (quieter least restrictive setting). We have tutoring support especially in Title 1 and Special Education where they work on specific skills, gaps, and skill deficits to help build that background so that they can remain in an inclusive classroom and receive core instruction. Sometimes, it’s not traditional. We are using an intervention aid where our kids check in early in the morning.”

Goal setting is a hallmark of MTSS programs in helping students develop academic goals based on their strengths and to create measurable markers in pursuit of that goal. Evidence based research shows the efficacy of those programs that focus on building relationships and increasing student engagement (Frazelle & Nagel, 2015). The point of the highest level, mentoring, involves relationship building and fostering a sense of belonging in the school community as well as providing structured support for academic goals. Respondents to the interview agree that the core of an intervention is relationship building to enhance student engagement. The end goal as shared by many respondents – to get students back on track so that much of their time can be spent with core instruction with less structured support. EWS data in high adoption schools was used to provide measures for this transition to core instruction and provide markers when to adjust or end an intervention.

Has EWS data inspired targeted interventions among participating schools?

Common EWS warning signs include missing 20 days or absent 10% of the school days, two or more behavior infractions, and inability to read at grade level, low GPA, or 2 or more failing grades in courses in the previous year (Knowles, 2015). Achievement data is not commonly used in these systems. This is because grades are seen as a stronger indicator as they reflect performance over time. According to interview respondents, by focusing attention on a small set of evidence-based indicators, users can conserve time and resources when managing and communicating the EWS data. By keeping it simple it is easier to define objective, evidence-based relationships and prioritize interventions.

In Montana, the core of the EWS predictive model addresses academics, attendance, behavior, and student mobility indicators. This is displayed to the user as a drop out probability and related risk factors. Schools receive this data in a summary for the school and a detail for each student. The policy decisions on how to identify students, how to frame thresholds of interventions, and the decision to use data to monitor students reflect local choices and local control. However, certain patterns of the intensity of interventions are apparent based on the use of the EWS model. The Montana EWS model has inspired targeted intervention among high adoption schools; however low adoption schools expressed interest in the model but did not implement it. There are questions whether these schools had policies that prioritized dropout prevention in the first place.

48.28% of survey respondents reported infrequently using EWS data, or any dropout prevention data, to plan and implement interventions. This indicates that often schools explored the EWS system but did not use the EWS data in intervention strategies. For many students no interventions were prescribed. Often analysis of the data was done by the school leader who may have acted on the data independent of an MTSS committee or used the data solely as a reference. In these cases, counselors and teachers often were not involved. The person tasked with managing the data was often the attendance secretary or IT support staff. Another insight of the intervention data is that 38% of educators report that their school provides interventions less than 50% of the time. This is another sign that among EWS adopters many schools did not offer dropout prevention services for high need students. It shines light on a nascent area of demand among low adoption schools to help inspire a renewed focus on dropout prevention. Many EWS users noted that the diagnostic data was important and gave them the chance to intervene early with students before they could be identified as having more risk. 69% of respondents reported that the intervention suggestions in the EWS reports were helpful. This included high adoption schools, which had both informal and formal dissemination practices.

With the students identified by the school, about half of the schools reported these students received EWS based support ‘often’ or ‘always.’ Among these high adoption schools there was intense engagement. Among all adopters there were other indications of the depth of a data culture. 79.3% of EWS inspired interventions receive a follow up. The depth that this follow up was informed by EWS data varied between high adopters -- in some data is used in a limited context by a small team. In these schools, teachers did not have access to the data.

Schools were hesitant to discuss the success of their processes. Many high adoption respondents cited that 75% of identified students went on to graduate or go on to the next grade. The efficiency data of the EWS system (2023) provides system wide trends. 80.37 percent of EWS students were identified as future graduates and indeed go on to graduate. 8.46% of EWS students were identified as dropouts and did go on to dropout. 5.16% of EWS students are identified as graduates, but they do dropout. 6.00% are identified as dropouts and eventually graduate. This 6% is both a sign of a potential inefficiency of the Montana EWS system and a sign of success. Many of these students received successful interventions and graduated instead of dropping out. By and large interview respondents expressed the need to adjust their processes and make them more relevant to the years after the pandemic. The tenor of the responses among high adopters focused on the need to reshape the data use model to address the remaining 25% that tends to have an ever-changing risk profile.

Framing the root causes and establishing intervention thresholds for students at risk of dropping out is challenging. Dropout rates are influenced not only by individual success in coursework and personal choices, but also by the context of schooling itself. Academic, attendance, and behavioral risk factors can either contribute to dropout or be turned into opportunities (Jerald, 2006; Bruce et al., 2011). Utilizing data-informed decisions, educators can strategically intervene and provide tiered supports to help students graduate. This is found with high adoption schools who made the most use of the data and tailored their school’s dropout response using the Montana EWS data.

The Montana Office of Public Instruction developed an EWS which contains data on dropout risk, probabilities, and related risk factors such as academics, attendance, behavior, and mobility. The OPI also provided support to schools focused on the range of interventions available, how to define a threshold, and when and in what ways to act on the data. Professional development occurred at both conferences and with targeted data coaching to high need schools. Successful implementation is seen when policy changes become apparent and serve to institutionalize practices. In Montana, this is characterized by the sharing of value, vision, and data to all stakeholders. This occurred in high adoption school models. In many cases in low adoption schools, the policy choice was not to intervene with at-risk students regardless of whether the choice was EWS data informed or not. This comments on entrenched policy choices in many of these schools and highlights factors in the demand for innovation. Schools that decided to implement the Montana EWS in their school had clear dropout prevention plans that framed data use.

Thresholds vary for the EWS indicators in different contexts (Knowles, 2015). In the case of Montana, these thresholds are established and monitored by schools who can tie the data to an intervention and follow data informed best practices. Often discussions about thresholds and the point to trigger a particular intervention is the focus of local MTSS processes. Within a system there is a tradeoff between accuracy, complexity, and identification of clear outcome measures (Ibid). Often this tradeoff comes at the point data is tied, or not, to an intervention. 48.28% of respondents said they use data to identify students less than 50% of the time, indicating that even when a decision is made to intervene, the decision does not necessarily include data in the process.

The focus of an Early Warning System is not just the data on risk factors or dropout probability. The crux of the system is what users decide to do with the data. Data use can occur in multiple stages from simply identifying potential non completers, to assigning interventions using locally defined thresholds, to monitoring interventions once in course, to reassigning interventions based on the available data. In addition, sometimes dropout prevention programs are more intensive in that there is grade level or school wide implementation of the program. Scale varies, although it is important to note that scale should meet the identified vision, value, demand, and capacity in the school for an EWS to be successful. We see this scale varying in Montana between high adoption schools. Perceptions of the vision and value of the data are similar with each of these levels. It is only upon the intensity of data informed practices that these two levels differ. This is seen acutely with the incidence of progress monitoring using the tool. It reflects differences in intensity and depth of the data culture. High adoption schools tend to use the EWS tool more frequently in progress monitoring (the decision to continue or end support), but not in all cases.

Communication is a key indicator of the degree of institutionalization of the EWS and dropout prevention. A school leader describes that communication is key to the process of assigning interventions. When designing early interventions, he describes how he talks to staff to get perspectives on each student’s circumstances then looks at the data and verify with staff as to the data accuracy. He engages the families in the process, something frequently done with the support of EWS data. Building relationships is important for him, for example, when finding a student, a mentor who is the right fit and defining which resources are available for each student. The goal is to increase student engagement by finding meaningful data informed supports.

Unfortunately, relatively few schools in our sample frequently engaged in mentoring. In addition, follow up was not frequently data informed. The intensity of student support was at times defined by universal interventions and the support that can be provided by a teacher in a classroom providing core instruction. High adoption schools tend to intervene at a greater depth than other kinds of adopters. The interventions include small group tutoring tasks or more intense one on one supports. The depth of these interventions is dependent on demand, capacity, and priorities. Many schools do not have the need to focus on dropout except in an informal capacity.

Unfortunately, some schools do not place an emphasis on dropout prevention. Ten respondents to the interview (n = 32) were not familiar with MTSS processes, including dropout prevention tied to behavioral strategies. When asked whether they focus on dropout, two of these schools had dropout prevention program grants from other sources which recently ended. There was little institutionalization of the lessons learned from the grants and follow through that consistently addresses dropout prevention. One finding from this study is that many schools are in the process of developing a data culture surrounding the MTSS process. In professional development activities, they claimed that OPI should continue to create a clear tie between the EWS and MTSS intervention strategies. Specific OPI support could include additional assistance on establishing local thresholds for triggering and monitoring an intervention.

In this article we focused on the tie of data to intervention. High adoption schools interviewed said vision from leadership pushed the model forward and their schools had made strides in the development of a data culture surrounding dropout prevention These schools all had a MTSS team or a school-based intervention team. In these schools there was value placed on addressing the issue of graduation, the diagnostic tool, and the ability to follow through with the intervention process. OPI support focused on these high adoption schools. Demand for innovation is still apparent among all kinds of EWS adopters. Renewed focus with high adoption schools and targeted support to low adoption schools based on the incorporation of MTSS models marks the way forward.

Participant Consent: This study was determined to be Exempt by the Montana State University Office of Research Compliance.

Acknowledgement

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305S210011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute, the U.S. Department of Education, or the Montana Office of Public Instruction.

Bruce, M., Horning Fox, J., & Balfanz, R. (2011). On Track for Success: The Use of Early Warning Indicator and Intervention Systems to Build a Grad Nation. https://eric.ed.gov/?id=ED526421.
Bruch, J., Gellar, J., Cattell, L., Hotchkiss, J., & Killewald, P. (2020). Using data from schools and child welfare agencies to predict near-term academic risks (REL 2020–027). Washington, DC: U.S. Department of Education, Institute of Edu-cation Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Mid-Atlantic. http://ies.ed.gov/ncee/edlabs.
Data Quality Campaign (2013). Using Early Warning Data to Keep Students on Track toward College and Careers: A Primer for State Policy Makers. https://dataqualitycampaign.org/resource/using-early-warning-data-keep-students-track-toward-college-careers/.
Faria, A.-M., Sorensen, N., Heppen, J., Bowdon, J., Taylor, S., Eisner, R., & Foster, S. (2017). Getting students on track for graduation: Impacts of the Early Warning Intervention and Monitoring System after one year (REL 2017–272). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest. Retrieved from https://files.eric.ed.gov/fulltext/ED573814.pdf.
Frazelle, S. & Nagel, A. (2015). A practitioner’s guide to implementing early warning systems (REL 2015–056). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. Retrieved from http://ies.ed.gov/ncee/edlabs.
Heppen, J. & Therriault, S.B. (2008). Developing Early Warning Systems to Identify Potential High School Dropouts. https://eric.ed.gov/?id=ED521558.
Jerald, C. (2006). Identifying Potential Dropouts: Key Lessons for Building an Early Warning Data System. https://eric.ed.gov/?id=ED499838.
Knowles, J. (2015). Of Needles and Haystacks: Building an Accurate Statewide Dropout Early Warning System in Wisconsin. Journal of Educational Data Mining 7(3). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/JEDM082
Koon, S., & Petscher, Y. (2015). Comparing methodologies for developing an early warning system: Classification and regression tree model versus logistic regression (REL 2015–077). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. http://ies.ed.gov/ncee/edlabs.
Marken, A., Scala, J, Husby-Slater, M. & Davis, G. (2020). Early Warning Intervention and Monitoring System Intervention Guide. American Institutes for Research: Chicago, IL. https://www.air.org/sites/default/files/EWIMS-Implementation-Guide-FINAL-July-2020.pdf.
O’ Cummings, M. & Therriault, S.B. (2015) From Accountability to Prevention: Early Warning Systems Put Data to Work for Struggling Students. American Institutes for Research: Chicago, IL. https://files.eric.ed.gov/fulltext/ED576665.pdf

No competing interests reported.

Download PDF

Editorial decision: Revision requested
01 Apr, 2024
Reviews received at journal
25 Mar, 2024
Reviewers agreed at journal
21 Mar, 2024
Reviewers agreed at journal
19 Mar, 2024
Reviewers invited by journal
15 Feb, 2024
Editor assigned by journal
15 Feb, 2024
Submission checks completed at journal
08 Feb, 2024
First submitted to journal
15 Nov, 2023

You are reading this latest preprint version

Early Warning Systems Achieve Their Goals When Data is Tied to an Intervention

Status:

Version 1

Abstract

Introduction

Background

Methods

Results

What policy changes occur in schools that have used EWS in their intervention model?

Demand

School Level Factors

Has EWS data inspired targeted interventions among participating schools?

Discussion and Conclusion

Declarations

References

Additional Declarations

Status:

Version 1