Implementation Strategies to Promote Measurement-based Care in Schools: Evidence from Mental Health Experts Across the United States

doi:10.21203/rs.3.rs-1044947/v2

Download PDF

Research

Implementation Strategies to Promote Measurement-based Care in Schools: Evidence from Mental Health Experts Across the United States

https://doi.org/10.21203/rs.3.rs-1044947/v2

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background: Despite an established, comprehensive taxonomy of implementation strategies, minimal guidance exists for how to select and adapt strategies to specific services and contexts. We employed a replicable method to identify the most feasible and important implementation strategies to increase mental health providers’ use of measurement-based care (MBC) in schools. MBC is the routine use of patient-reported progress measures throughout treatment to inform patient-centered, data-driven treatment adjustments.

Methods: A national sample of 52 school mental health providers and researchers completed two rounds of modified Delphi surveys to rate the relevance, importance, and feasibility of 33 implementation strategies identified for school settings. Strategies were reduced and definitions refined using a multimethod approach. Final importance and feasibility ratings were plotted on “go-zone” graphs and compared across providers and researchers to identify top-rated strategies.

Results: The initial 33 strategies were rated as “relevant” or “relevant with changes” to MBC in schools. Importance and feasibility ratings were high overall for both survey rounds; on a scale of 1 to 5, importance ratings (3.61 - 4.48) were higher than feasibility ratings (2.55 – 4.06) on average. Survey 1 responses resulted in a reduced, refined set of 21 strategies, and six were rated most important and feasible on Survey 2: 1) assess for readiness and identify barriers and facilitators; 2) identify and prepare champions; 3) develop a usable implementation plan; 4) offer a provider-informed menu of free, brief measures; 5) develop and provide access to training materials; and 6) make implementation easier by removing burdensome documentation tasks. Provider and researcher ratings were not significantly different, with a few exceptions: providers reported higher feasibility and importance of removing burdensome paperwork than researchers, and providers reported higher feasibility of train-the trainer approaches than researchers; researchers reported higher importance of monitoring fidelity than providers.

Conclusions: The education sector is the most common setting for child and adolescent mental health service delivery in the United States. Effective MBC implementation in schools has the potential to elevate the quality of care received by many children, adolescents, and their families. This empirically-derived, targeted list of six implementation strategies offers potential efficiencies for future testing of MBC implementation in schools.

implementation strategy selection

measurement-based care

school mental health treatment

Delivering mental health treatment services in schools improves access for children and adolescents, but school context-specific implementation strategies are needed to increase best practices in this setting.
Measurement-based care (MBC) is a scalable, evidence-based clinical practice that can be added to nearly any school-based mental health treatment approach, but school systems, administrators and providers need targeted supports to increase MBC implementation.
The most important and feasible strategies to implement MBC in schools focus on training and supporting mental health providers plus addressing systemic barriers such as access to measures and burdensome documentation.
Stakeholders agreed on strategy feasibility and importance with some exceptions; providers emphasized the importance and feasibility of removing burdensome paperwork and researchers emphasized the importance of fidelity monitoring.
We used an established, stakeholder-engaged method to systematically select and tailor implementation strategies that can be replicated for other practices and settings.

Measurement-based care (MBC) is the routine collection and use of client data throughout treatment, including initial screening and assessment, problem definition and analysis, finalizing treatment objectives and intervention tactics, and monitoring treatment progress collaboratively with the client to inform mid-course treatment plan adjustments (31). MBC is a critical component of an evidence-based practice orientation to mental health treatment (32). There is strong evidence supporting MBC in treatment settings other than schools. For instance, systematic reviews show better and faster goal attainment and symptom reduction with MBC as compared to usual care; effect sizes range from 0.28 to 0.70 (33–35). Larger effect sizes of 0.49 to 0.70 are attributed to MBC with feedback, particularly feedback provided to both the patient and providers, or when clinical support tools are provided (35, 36). Recent Cochrane reviews underscore the importance of including studies where measures are used to adjust the treatment plan (37, 38), indicating that patient outcomes associated with MBC are likely a result of the real time, client-centered, data-driven adjustments made to interventions provided.

Despite the promise of MBC to improve mental health service quality, use of MBC in practice is minimal. Fewer than 20% of providers report collecting progress measures at least monthly (39, 40). Barriers to MBC implementation in behavioral health care have been well-documented at the individual patient, provider, organizational, and system levels (41). These include concerns about time and administrative burden, provider attitudes toward MBC as no better than clinical judgment alone, concern about how data will be used, and limited provider training resources or incentives to implement (41). Yet, when providers use clinical judgment alone without a structured assessment process, they are much less accurate in predicting progress and risk, providing services that are neither individualized to patient concerns nor responsive to progress (42–44).

Participants

Study participants (N=52) were drawn from a national sample of school mental health stakeholders: 1) School-based providers with experience providing and/or supervising mental health interventions in schools (N=31); and 2) researchers with experience partnering with schools or districts to implement EBPs (N=21). Providers were sampled from the National School Mental Health Census and researchers were sampled from their established list of 138 school mental health researchers. All participants are US-based, and work in 23 states (AZ, AR, CA, CO, CT, FL, GA, IL, IN, LA, MD, MA, MI, MN, NE, NH, NC, OH, OR, PA, TX, VA, and WA).

Demographic, professional, and urbanicity characteristics of participants are in Table 1. School providers identified primarily as school psychologists (N = 6, 19%), school social workers (N = 5, 16%), or school counselors (N = 5, 16%). Other provider roles included being a school psychology supervisor (N = 2, 7%), director of related services/special education/student support (N = 2, 7%), counselor (community- or hospital-employed; N = 1, 3%), mental health agency administrator (N = 1, 3%) or other position (N = 9, 29%). School-based providers worked in 18 states representing all regions of the United States, and researchers worked in 14 states and worked with school partners in 43 states at the time of data collection or in the past, the District of Columbia, Guam, the US Virgin Islands and other US territories. Most School-based providers indicated they had current or past experiences providing (N = 30, 97%) and/or supervising (N = 20, 65%) mental health treatment services in schools. All researchers had experience conducting research about child and adolescent mental health, conducting research in partnership with schools/districts, training school-based personnel, and providing consultation or technical assistance to schools/districts. Most researchers had current or past experience training graduate students about working in or with schools (N = 20, 95%), providing mental healthcare in schools (N = 16, 77%), supervising direct mental healthcare in schools (N = 13, 62%) and serving as an educator (N = 11, 52%). There was a 94% retention rate of participants for Survey 2 (N=49; N=30 providers and N=19 researchers). Demographic and professional characteristics and urbanicity of participating providers were representative of those recruited (70). Researchers represented various age groups, field of training and urbanicity across the United States, and gender identity (56% Female) and degree (100% PhD) was very similar to researchers in our datasets who were not invited to participate. Thus, results from participants in this study are likely to be generalizable to stakeholders of similar demographics, professional expertise, and geographic location.

Procedures

Systematic sampling procedures that drew on nationally-representative databases for school-based providers and researchers were used to identify the study sample. Providers were selected through random stratified sampling from the National School Mental Health Census, a nationally-representative survey of school and district mental health teams’ services and data usage. Inclusion criteria was holding a position as a school mental health provider or clinical supervisor with likelihood of experience delivering or supervising school-based psychotherapy, in which MBC would be used (e.g., school social worker). Census data with individuals meeting this inclusion criteria were stratified based on rural-urbanicity continuum codes (metropolitan vs. non-metropolitan) and geographic representation. Prospective participants were randomly selected with replacement until a target sample of at least 30 school mental health providerswas achieved. We monitored the sample for approximate distributions in the United States for 1) metropolitan and non-metropolitan/rural urbanicity; and 2) geographic location. Using this approach, we oversampled for non-metropolitan/rural providers toward the end of recruitment to ensure adequate representation.

We recruited 194 school mental health providers; after a response rate of 19% (N=36), five were ineligible and thus screened out for a final sample of 31 providers. Eligible recruits who did not participate had nonworking emails (N=24), did not respond to our recruitment request (N=106), or declined (N=28). Providers received up to three reminder emails over the course of three weeks to respond to the study invitation to consent and start Survey 1.

Researchers were selected using purposive sampling from two sources, which were 1) Implementation Research Institute fellows who applied to and were selected for implementation science training via a competitive process were reviewed for school mental health expertise (71); and 2) an established list maintained by the National Center for School Mental Health of 138 school mental health researchers with active peer-reviewed publications and/or grants on topics pertaining to school mental health and wellbeing. This latter group of researchers were part of an invitation-only annual meeting and pre-reviewed for their scholarship and impact on the field, adjusted for career stage, by a planning committee team comprised of national school mental health scholars. Inclusion criteria were: 1) expertise with mental health program or practice development, effectiveness testing and/or implementation research; 2) experience partnering directly with schools; and 3) Associate Professor or Professor at their institution, which resulted in N=56 eligible researchers. Next, advanced expertise implementing mental health programs or practices in schools was coded on a four-point scale (3 = “optimal”, 2 = “good”, 1 = “okay’, and 0 = “unable to assess”) by three senior school mental health researchers with extensive experience in evidence-based practice implementation in schools. Ratings were averaged for each researcher and then recruits were invited with replacement from the highest ratings downwards until a sample size of at least N=20 was achieved. We recruited 29 research participants, which resulted in a response rate of 72% (N=21); among recruits, one did not respond to recruitment emails and seven declined.

Measures: Delphi Surveys

Participants completed two rounds of feedback using anonymous Delphi surveys. Each survey started with operational definitions of implementation strategies, MBC, school mental health providers, and three vignettes illustrating MBC use in schools (see Supplemental File 1). Vignettes were developed and revised for clarity and accuracy based on feedback from several co-authors and other collaborators. The vignettes intentionally focus on MBC clinical practice representing various school mental health professional roles, presenting concerns, student ages[1], and measures. Due to our focus on identifying implementation strategies for MBC as a clinical practice, the vignettes did not refer to any implementation supports, such as decision support by a measurement feedback system or other digital interface for scoring and viewing progress data.

The Delphi technique is an established method using a series of surveys or questionnaires to obtain controlled, mixed methods input from a diverse set of expert stakeholders so as to gain reliable consensus on a health care quality topic (72, 73). This method was used in the Expert Recommendations for Implementing Change (ERIC) project to identify a complete list of implementation strategies and definitions for selection and application to specific practices and settings (8, 68). Another research team then replicated and extended this research to identify an adapted set of important and feasible strategies for implementing evidence-based practices in schools (27, 74, 75). The Round 1 survey included 33 implementation strategies rated most important and feasible by a prior study examining evidence-based practices generally in schools (74). For each strategy, participants indicated whether it is relevant to MBC care specifically (“yes”, “yes with changes” or “no”). For strategies rated as relevant (“yes” or “yes with changes”), participants then were asked to provide 1) importance and feasibility ratings (1 = “not at all important/feasible” to 5 = “extremely important/feasible”), 2) possible synonyms or related activities to the strategy, and 3) suggestions about the definition or application of the strategy. At the end of the survey, participants were also asked to suggest additional implementation strategies not listed. The Round 2 survey included an updated list of strategies and definitions based on Round 1 results. Participants had four weeks to complete Round 1 and 2 surveys. Participants were compensated for their time and study procedures were approved by the Yale Institutional Review Board.

Data Analyses

Descriptive statistics of quantitative feasibility and importance ratings were examined for normality. Independent samples t-tests were used to compare ratings between providers and researchers. Mean feasibility and importance ratings were plotted for each strategy on a “go-zone” plot to compare relative feasibility and importance by quadrants (76). Go-zone plots provide a bivariate display of mean ratings and are often used in concept mapping. The origin represents the grand mean of both variables of interest (in this case, feasibility and importance) and the four resulting quadrants are used to interpret relative distance among items (in this case, strategies). The top right quadrant is the “go-zone” where strategies of the highest feasibility and importance appear.

A multimethod approach was used to reduce strategies and refine definitions between Survey 1 and Survey 2. First, a document was developed to display quantitative and qualitative Survey 1 results for each strategy. This included each Survey 1 strategy and definition, go-zone quadrant results (overall, as well as for providers and researchers), quantitative considerations (e.g., percentage of stakeholders who indicated the strategy was not relevant for MBC in school, significant differences between providers and researchers, any distribution normality concerns with ratings), qualitative synonyms, and qualitative definition change recommendations made by participants. Second, one rater (EC) reviewed each strategy using this document and established decision-making guidance vetted by study team members for each go-zone. She coded an initial decision (e.g., retain with revisions, collapse, or remove) with justification for each, documented any synonyms reported more than three times and drafted definition changes that were a) minimal language adjustments; b) not substantial additions to definition length; and b) consistent with overall scope of the strategy. Then, another rater (CS) reviewed coded decisions and documentation, and all discrepancies were resolved through consensus conversations. Final decisions about collapsing strategies were made based on consultation with two implementation researchers.

To analyze Survey 2 results, descriptive statistics, independent samples t-tests and go-zone plots were used again, as was the multi-step process detailed above.

[1] Vignettes refer to student “grade”, not age. In the United States, 3^rd grade is in primary school, approximately 8 years old, 6^th grade is considered “middle school”, approximately 11 years old, and 9^th grade students is the beginning of secondary school, approximately 14 years old.

Survey 1 Strategy Ratings

In general, strategies were rated as “relevant” or “relevant with changes” by participants, and all 33 strategies in Survey 1 received importance and feasibility ratings. Eight strategies received the highest proportion of “not relevant” ratings (range = 25-38% participants) to MBC in schools, as follows: 1) model and simulate change; 2) change/alter environment; 3) provide practice-specific feedback; 4) identify early adopters; 5) visit other sites; 6) obtain and use student and family feedback; 7) develop academic partnerships; and 8) build partnerships (i.e., coalitions) to support implementation. However, as the majority of participants rated these as “relevant” or “relevant with changes”, the importance and feasibility ratings are included in our analysis.

Importance and feasibility ratings were high overall for both survey rounds, with importance ratings higher than feasibility ratings on average. On Survey 1, importance ratings ranged from 3.44 (“develop academic partnerships”) to 4.61 (“make implementation easier by removing burdensome documentation tasks”) and feasibility ratings ranged from 2.89 (“visit other sites”) to 4.10 (“distribute educational materials”). Survey 1 standard deviations of mean scores varied from 0.68 to 1.18. See Table 2 for importance and feasibility results for the 33 initial implementation strategies. Figures 1 and 2 display these findings on go-zone plots, where the four quadrants or “zones” are divided by the grand mean scores of 4.01 for importance and 3.49 for feasibility. Zone 1 includes strategies rated above the grand mean for importance and feasibility (i.e., high feasibility/high importance), Zone 2 includes strategies rated above the grand mean for feasibility but not importance (i.e., high feasibility/low importance), Zone 3 includes strategies rated below the grand mean for feasibility and importance (i.e., low feasibility/low importance), and Zone 4 includes strategies rated above the grand mean for importance but below the feasibility grand mean (i.e., low feasibility/high importance).

Survey 2 Strategy Ratings

Based on the multimethod approach described above, Survey 2 contained a reduced set of 21 strategies with updated definitions (see Figure 3). From Survey 1 to Survey 2, 14 strategies were retained (with updates to strategy title and/or definition in most cases), 7 strategies were collapsed into three, 12 were removed and 4 were added. Feasibility and importance grand means were similar for Survey 2 (importance grand mean = 4.05; feasibility grand mean = 3.33). On Survey 2, importance ratings ranged from 3.61 (“use train the trainer strategies”) to 4.48 (“develop a useable implementation plan”) and feasibility ratings ranged from 2.55 (“support workflow adjustments”) to 4.06 (“offer a provider-informed menu of free, brief measures”). Survey 2 standard deviations of mean scores varied from 0.56 to 1.22.

Survey 2 Top-Rated Strategies

Among the 21 revised implementation strategies included in Survey 2 (see Table 4), six were rated as most important and most feasible (see go-zone 1 strategies in Table 3, Figure 3 and Figure 4). These top-rated strategies include: 1) Assess for readiness and identify barriers and facilitators; 2) identify and prepare champions; 3) develop a usable implementation plan; 4) offer a provider-informed menu of free, brief measures; 5) develop and provide access to training materials; and 6) make implementation easier by removing burdensome documentation tasks. Of note, these are all strategies that would need to be arranged by administrators or other leaders organizing an implementation approach for their clinical providers. Also, there is a natural chronology to how these six strategies might be combined; several strategies would need to be put in place before an initial training or provision of training materials occurs (e.g., assess for readiness, develop an implementation plan). That said, these strategies are all relatively distinct and could be reasonably provided as a “bundle” of strategies to support MBC implementation in schools.

Several additional strategies were rated within 0.50 of the feasibility grand mean, yet above the mean cutoff for importance (see Table 3, go-zone 4 strategies with asterisks). These include “conduct ongoing training”, “provide ongoing clinical consultation/coaching”, “monitor implementation progress and provide feedback”, “monitor fidelity to MBC core components” and “promote adaptability”. In general, these strategies reflect those which promote ongoing implementation in clinical practice after initial planning and provider training, which is highly consistent with extant findings about the importance of post-training implementation support strategies (77–79). As these strategies are near the “border” of feasibility and importance grand means (i.e., nearer the grand mean rating than the next point higher on the 5-point Likert scale), they warrant attention as potentially-viable strategies, given the strictly numeric, bivariate cutoff between zones based on grand mean values.

Stakeholder Group Comparisons

On Survey 1, provider and researcher ratings were not significantly different with three exceptions. First, as compared to researchers, providers reported that it is more feasible and important to make implementation easier by removing burdensome paperwork (feasibility t(44) = -2.96, p = 0.01; importance t(29) = -2.72, p = 0.01). Second, as compared to providers, researchers reported it is more important to monitor the implementation effort (t(41) = 2.51, p = 0.02). Third, train-the-trainer feasibility ratings were significantly higher among providers than researchers (t(45) = -2.06, p = 0.05). On Survey 2, provider and researcher ratings were not significantly different with one exception; providers reported it is more important to make implementation easier by removing burdensome paperwork (t(44) = 2.04, p = 0.048).

We applied an established, stakeholder-informed method to identify the most important and feasible implementation strategies for measurement-based care (MBC) used in school-based mental health treatment. MBC was selected as an under-implemented yet promising and scalable clinical practice in schools that can be added to any presenting concern or treatment plan to improve care quality for children and adolescents. We identified six top-rated implementation strategies for MBC based on ratings of importance and feasibility in schools. Those strategies were: 1) assess for readiness and identify barriers and facilitators; 2) identify and prepare champions; 3) develop a usable implementation plan; 4) offer a provider-informed menu of free, brief measures; 5) develop and provide access to training materials; and 6) make implementation easier by removing burdensome documentation tasks.

Several additional strategies related to promoting ongoing implementation in clinical practice after initial planning and provider training were rated as highly important and relatively feasible within 0.50 of the feasibility grand mean. Providers and researchers participated in the study to provide a diversity of perspectives on implementation strategies. In general, stakeholders’ ratings were consistent across stakeholder groups, although providers emphasized the reduction of burdensome documentation and researchers emphasized fidelity monitoring to support MBC in schools. These differences have face validity; providers have more experience with barriers related to documentation and other clinical workflow details than researchers do, and researchers are more focused on ensuring the implementation is carried out as intended by the research design. This difference also points to the importance of ensuring bidirectional communication, collaboration, and perspective sharing between those on the “front lines” of clinical implementation and implementation researchers. These divergent findings also lend credibility to the approach of sampling various stakeholder perspectives when examining implementation processes.

As mentioned earlier, this study applied an existing methodology used in prior research to identify the most important and feasible implementation strategies for a given context or innovation. Compared to prior research by (69) to identify implementation strategies for any evidence-based practice in schools, our results focused specifically on MBC implementation in schools. The current results reflect a narrower and higher range of both importance and feasibility ratings as compared to general EBP implementation in schools’ importance range (our importance range = 3.61 – 4.48 versus a range of 2.62 – 4.59 in prior work and our feasibility range = 2.55 – 4.06 versus a range of 2.08 – 3.72 in prior work. This is not surprising as we selected the 33 strategies in SISTER go-zone 1 (i.e., those rated most important and feasible for evidence-based practices generally implemented in schools) for inclusion in our Survey 1. The foundation of the prior use of this implementation strategy selection method was a critical platform for this work in order to focus our efforts on a smaller set of implementation strategies, specifically important and feasible in the school context, as compared to the initial set of 73 strategies identified by (8). In turn, our study has further reduced the set of implementation strategies most likely to be important and feasible in schools for a specific evidence-based practice, MBC. This data reduction process for targeted implementation strategy selection is highly relevant for implementation providers, who cannot possibly apply the universe of potential strategies to a given implementation effort, nor is there yet sufficient empirical evidence on which strategies to apply for a specific practice or setting.

Top-rated strategies from these results span provider- and organizational-level approaches to improve implementation. This is consistent with extant literature suggesting that implementation barriers to evidence-based practices generally at the provider and school system levels are well-documented (45, 80–82). Intervention-level and outer context strategies are generally underdeveloped (26, 83). Thus, the research-practice gap in implementation science could be addressed in part by further identification and specification of implementation strategies to pragmatically target multilevel determinants.

Contextually-tailored implementation strategies are believed to optimize intervention-setting fit and implementation outcomes. For example, offering a “provider-informed menu of free, brief measures” was an MBC-specific implementation strategy recommended by participants in this study who are highly familiar with clinical care delivered in school settings. This strategy is specific to MBC practices in schools, where providers often have limited access to brief, validated measures they desire to use (84).

This study has several limitations. First, although this sample was nationally representative, it is relatively small, and thus importance and feasibility ratings may not hold for a larger sample. Also, school providers were recruited from a national dataset of teams engaged in school mental health quality assessment and improvement efforts, which may be a more select group of school mental health providers who are more invested in in advancing school mental health quality and/or use of psychosocial data in school mental health treatment. Future studies should examine importance and feasibility ratings from a wider range of school mental health providers from the general population of school stakeholders. Also, we selected 33 implementation strategies already rated highly in a prior study of EBP implementation in schools, and reduced to six strategies based on further high ratings, so we were unlikely to find mean importance or feasibility ratings in the low to moderate range. Although this may raise questions about potential ceiling effects, the grand means for each construct were not overly high (importance grand mean = 4.05; feasibility grand mean = 3.33) and we used the grand mean as the cut point for the sample (as is conventional for go-zone graphs) to interpret differences among ratings. Finally, stakeholders’ qualitative feedback about the definition of each strategy was used to develop the final list that appears in Table 4, but recommendations about application of the strategy were not included. This is most pertinent to feasibility, and our team is currently examining these qualitative data to understand how we might optimize feasibility of individual strategies (e.g., provide ongoing clinical consultation/coaching) that were rated highly important, but relatively less feasible (70). Feasibility is a complex construct; many elements contribute to feasibility ratings for a given practice or strategy (85) and when we assess perceptions of feasibility prospectively, the rater has to make assumptions about what resource or training requirements, for example, are part of the strategy (6). It is not uncommon for school stakeholders to rate implementation supports or best practices as more important than feasible due to their experience with resource constraints and structural barriers in schools (30, 86). Therefore, future research should continue to examine how to operationalize, tailor and evaluate strategies to promote feasibility in the school context, in order to support schools’ capacity to feasibly implement new initiatives with integrity and sustainability (45, 87).

Methods to select and tailor implementation strategies for a particular practice and setting have been somewhat elusive to date in implementation research and practice (4). The methods used in this study can be applied to other evidence-based practices, settings, and contexts to answer practice-specific implementation challenges in other service settings. For example, the current research could be replicated to identify the most important and feasible implementation strategies for MBC in other settings, or for other evidence-based practices in school settings. Our sample included both provider and researcher stakeholders across the US, which we recommend to ensure multiple and diverse perspectives are represented in future research.

In addition, implementation strategies selected for their potential importance and feasibility need to be further examined in application. Identification of top-rated strategies for a particular intervention and context is foundational to future strategy testing with practicing providers in real world care systems. Strategies selected from implementation science methods, such as the current survey methods with go-zone plots, should also be critically examined for the possibility of bundling or combining some strategies together (for parsimony and/or alignment) as well as when to apply strategies across implementation stages over time.

MBC

Measurement-Based Care

SISTER

School Implementation Strategies, Translating ERIC Resources (SISTER) Project

EBP

Evidence-based practice

Ethics approval and consent to participate

All study procedures were reviewed and approved by the Yale University institutional review board. The study team conducted informed consent meetings via phone with prospective participants and received written/verbal consent consistent with IRB-approved procedures.

Consent for publication

Not applicable.

Availability of data and material

The datasets generated during and/or analyzed during the current study are not publicly available due to the pilot nature of the study and lack of patient outcome data but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This publication was funded by the National Institute of Mental Health (K08MH116119). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author contributions

EHC developed the overarching scientific aims and design of the project. EHC led all aspects of the study under close mentorship and guidance of JKT. ARL provided study design and methodological consultation specifically related to survey construction and go-zone plot analyses. ARL, SH, MW and JKT supported the design and execution of participant recruitment. KG consented study participants. EHC, KG and CS were involved in data collection, management and analysis with consultation and input from all other co-authors. All authors contributed to the development, drafting, or review of the manuscript. All authors approved the final manuscript.

Acknowledgements

We are grateful to Drs. Nancy Lever and John Landsverk for their consultation, support and review of study design and materials. We also acknowledge the role of The SHAPE System (www.theshapesystem.com) as a national quality improvement system for school mental health in the United States from which provider participants were recruited.

Drake RE, Goldman HH, Leff HS, Lehman AF, Dixon L, Mueser KT, et al. Implementing evidence-based practices in routine mental health service settings. Psychiatric services. 2001;52(2):179-82.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science. 2009;4(1):50.
Lyon AR, Ludwig K, Romano E, Koltracht J, Vander Stoep A, McCauley E. Using modular psychotherapy in school mental health: Provider perspectives on intervention-setting fit. Journal of Clinical Child & Adolescent Psychology. 2014;43(6):890-901.
Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the impact of implementation strategies in healthcare: a research agenda. Frontiers in public health. 2019;7:3.
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):24-34.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(2):65-76.
Waltz TJ, Powell BJ, Chinman MJ, Smith JL, Matthieu MM, Proctor EK, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implementation Science. 2014;9(1):1-12.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science. 2015;10(1):21.
Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to improve the selection and tailoring of implementation strategies. The journal of behavioral health services & research. 2017;44(2):177-94.
Sanetti LMH, Collier-Meek MA. Increasing implementation science literacy to address the research-to-practice gap in school psychology. Elsevier; 2019.
Olswang LB, Prelock PA. Bridging the gap between research and practice: Implementation science. Journal of Speech, Language, and Hearing Research. 2015;58(6):S1818-S26.
Wensing M, Grol R. Knowledge translation in health: how implementation science could contribute more. BMC medicine. 2019;17(1):88.
Lyon AR, Comtois KA, Kerns SE, Landes SJ, Lewis CC. Closing the science–practice gap in implementation before it widens. In: Albers BS, Aron; Mildon, Robyn, editor. Implementation Science 302020. p. 295-313.
Westerlund A, Sundberg L, Nilsen P. Implementation of Implementation Science Knowledge: The Research-Practice Gap Paradox. Worldviews Evid Based Nurs. 2019;16(5):332-4.
Dariotis JK, Mirabal-Beltran R, Cluxton-Keller F, Feagans Gould L, Greenberg MT, Mendelson T. A Qualitative exploration of implemenation factors in a school-based mindfulness and yoga program: Lessons learned from students and teachers. . Psychology in the Schools. 2017;54(1):53-69.
Landes SJ, Rodriguez AL, Smith BN, Matthieu MM, Trent LR, Kemp J, et al. Barriers, facilitators, and benefits of implementation of dialectical behavior therapy in routine care: results from a national program evaluation survey in the Veterans Health Administration. Translational Behavioral Medicine. 2017;7(4):832-44.
Lindamer LA, Lebowitz B, Hough RL, Garcia P, Aguirre A, Halpain MC, et al. Establishing an implementation network: lessons learned from community-based participatory research. Implementation Science. 2009;4(1):17.
Ploeg J, Davies B, Edwards N, Gifford W, Miller PE. Factors Influencing Best-Practice Guideline Implementation: Lessons Learned from Administrators, Nursing Staff, and Project Leaders. Worldviews on Evidence-Based Nursing. 2007;4(4):210-9.
Capaldo G, Rippa P. A planned‐oriented approach for EPR implementation strategy selection. Journal of Enterprise Information Management. 2009.
Highfield L, Valerio MA, Fernandez ME, Eldridge-Bartholomew LK. Development of an Implementation Intervention Using Intervention Mapping to Increase Mammography Among Low Income Women. Frontiers in Public Health. 2018;6(300).
Bains RM, Diallo AF. Mental health services in school-based health centers: systematic review. The Journal of School Nursing. 2016;32(1):8-19.
Farmer EM, Burns BJ, Phillips SD, Angold A, Costello EJ. Pathways into and through mental health services for children and adolescents. Psychiatric services. 2003;54(1):60-6.
Duong MT, Bruns EJ, Lee K, Cox S, Coifman J, Mayworm A, et al. Rates of Mental Health Service Utilization by Children and Adolescents in Schools and Other Common Service Settings: A Systematic Review and Meta-Analysis. Administration and Policy in Mental Health and Mental Health Services Research. 2021;48(3):420-39.
Cummings JR, Ponce N, Mays VJTJoahopotSfAM. Comparing racial/ethnic differences in mental health service use among high-need subpopulations across clinical and school-based settings. 2010;46 6:603-6.
Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implementation Science. 2015;10(1):1-8.
Dopp AR, Parisi KE, Munson SA, Lyon AR. Integrating implementation and user-centred design strategies to enhance the impact of health services: protocol from a concept mapping study. Health research policy and systems. 2019;17(1):1-11.
Cook CR, Lyon AR, Locke J, Waltz T, Powell BJ. Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science. 2019;20(6):914-35.
Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ open. 2021;11(10):e053474.
Ageberg E, Bunke S, Lucander K, Nilsen P, Donaldson A. Facilitators to support the implementation of injury prevention training in youth handball: a concept mapping approach. Scandinavian journal of medicine & science in sports. 2019;29(2):275-85.
Stormont M, Lewis TJ, Covington Smith S. Behavior support strategies in early childhood settings: Teachers' importance and feasibility ratings. Journal of Positive Behavior Interventions. 2005;7(3):131-9.
Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cognitive and behavioral practice. 2015;22(1):49-59.
Resnick SG, Oehlert ME, Hoff RA, Kearney LK. Measurement-based care and psychological assessment: Using measurement to enhance psychological treatment. Psychological Services. 2020;17(3):233-7.
Christian U. Krägeloh, Ph.D. ,, Karol J. Czuba, M.Phty. ,, D. Rex Billington, Ph.D. ,, Paula Kersten, Ph.D. ,, Richard J. Siegert, Ph.D. Using Feedback From Patient-Reported Outcome Measures in Mental Health Services: A Scoping Study and Typology. Psychiatric Services. 2015;66(3):224-41.
Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta‐analysis. Clinical Psychology: Science and Practice. 2003;10(3):288-301.
Lambert MJ, Whipple JL, Kleinstäuber M. Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy. 2018;55(4):520.
Shimokawa K, Lambert MJ, Smart DW. Enhancing treatment outcome of patients at risk of treatment failure: meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of consulting and clinical psychology. 2010;78(3):298.
Bergman H, Kornør H, Nikolakopoulou A, Hanssen‐Bauer K, Soares‐Weiser K, Tollefsen TK, et al. Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews. 2018(8).
Kendrick T, El‐Gohary M, Stuart B, Gilbody S, Churchill R, Aiken L, et al. Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews. 2016(7).
Bickman L, Rosof-Williams J, Salzer M, Summerfelt W, Noser K, Wilson S, et al. What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice. 2000;31:70-4.
Jensen-Doss A, Haimes EMB, Smith AM, Lyon AR, Lewis CC, Stanick CF, et al. Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research. 2018;45(1):48-61.
Lewis CC, Boyd M, Puspitasari A, Navarro E, Howard J, Kassab H, et al. Implementing Measurement-Based Care in Behavioral Health: A Review. JAMA Psychiatry. 2019;76(3):324-35.
Hatfield D, McCullough L, Frantz SH, Krieger K. Do we know when our clients get worse? An investigation of therapists' ability to detect negative client change. Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice. 2010;17(1):25-32.
Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DW, Shimokawa K, et al. A lab test and algorithms for identifying clients at risk for treatment failure. Journal of clinical psychology. 2005;61(2):155-63.
Henke RM, Zaslavsky AM, McGuire TG, Ayanian JZ, Rubenstein LV. Clinical inertia in depression treatment. Medical care. 2009;47(9):959.
Langley AK, Nadeem E, Kataoka SH, Stein BD, Jaycox LH. Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School mental health. 2010;2(3):105-13.
Barrett NM, Gill KJ, Pratt CW, Roberts MM. Psychiatric rehabilitation: Academic Press; 2013.
Lindsey MA, Chambers K, Pohle C, Beall P, Lucksted A. Understanding the behavioral determinants of mental health service use by urban, under-resourced Black youth: Adolescent and caregiver perspectives. Journal of Child and Family Studies. 2013;22(1):107-21.
Whitaker K, Nicodimos S, Pullmann MD, Duong MT, Bruns EJ, Wasse JK, et al. Predictors of Disparities in Access and Retention in School-Based Mental Health Services. School Mental Health. 2018;10(2):111-21.
Graczyk PA, Domitrovich CE, Zins JE. Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. Handbook of School Mental Health Advancing Practice and Research: Springer; 2003. p. 301-18.
Tebes JK, Feinn R, Vanderploeg JJ, Chinman MJ, Shepard J, Brabham T, et al. Impact of a Positive Youth Development Program in Urban After-School Settings on the Prevention of Adolescent Substance Use. Journal of Adolescent Health. 2007;41(3):239-47.
Hagermoser Sanetti LM, Collier-Meek MA. Increasing implementation science literacy to address the research-to-practice gap in school psychology. Journal of School Psychology. 2019;76:33-47.
Lim CT, Fulwiler CE, Carson NJ, Huang H, Robinson LA, Schuman-Olivier Z, et al. Promoting Population Behavioral Health in a Safety-Net Health System During the COVID-19 Pandemic. Psychiatr Serv. 2021;72(10):1225-8.
Fitzpatrick O, Carson, A. & Weisz, J.R. Using Mixed Methods to Identify the Primary Mental Health Problems and Needs of Children, Adolescents, and Their Caregivers during the Coronavirus (COVID-19) Pandemic. . Child Psychiatry & Human Development. 2020;52:1082-93.
Joosen MC, van Beurden KM, Terluin B, van Weeghel J, Brouwers EP, van der Klink JJ. Improving occupational physicians’ adherence to a practice guideline: feasibility and impact of a tailored implementation strategy. BMC medical education. 2015;15(1):1-12.
Engel M, Bruns A, Hulscher M, Gaillard C, Sankatsing S, Teding van Berkhout F, et al. A tailored implementation strategy to reduce the duration of intravenous antibiotic treatment in community-acquired pneumonia: a controlled before-and-after study. European journal of clinical microbiology & infectious diseases. 2014;33(11):1897-908.
Wensing M. The Tailored Implementation in Chronic Diseases (TICD) project: introduction and main findings. Implementation Science. 2017;12(1):1-4.
Owens JS, Lyon AR, Brandt NE, Warner CM, Nadeem E, Spiel C, et al. Implementation science in school mental health: Key constructs in a developing research agenda. School mental health. 2014;6(2):99-111.
Weist MD, Youngstrom EA, Stephan S, Lever N, Fowler J, Taylor L, et al. Challenges and ideas from a research program on high-quality, evidence-based practice in school mental health. Journal of Clinical Child & Adolescent Psychology. 2014;43(2):244-55.
Connors EH, Arora P, Curtis L, Stephan SH. Evidence-Based Assessment in School Mental Health. Cognitive and Behavioral Practice. 2015;22:60-73.
Ruble L, McGrew JH, Toland MD. Goal attainment scaling as an outcome measure in randomized controlled trials of psychosocial interventions in autism. Journal of autism and developmental disorders. 2012;42(9):1974-83.
Miller FG, Crovello NJ, Chafouleas SM. Progress monitoring the effects of daily report cards across elementary and secondary settings using Direct Behavior Rating: Single Item Scales. Assessment for Effective Intervention. 2017;43(1):34-47.
Connors E, Lawson G, Wheatley-Rowe D, Hoover S. Exploration, Preparation, and Implementation of Standardized Assessment in a Multi-agency School Behavioral Health Network. Administration and Policy in Mental Health and Mental Health Services Research. 2020:1-18.
Sander MA, Everts J, Johnson J. Using data to inform program design and implementation and make the case for school mental health. Advances in School Mental Health Promotion. 2011;4(4):13-21.
Detterman R, Ventura J, Rosenthal L, Berrick K. Unconditional Education: Supporting Schools to Serve All Students: Oxford University Press; 2019.
Fletcher JM, Vaughn S. Response to intervention: Preventing and remediating academic difficulties. Child development perspectives. 2009;3(1):30-7.
Borntrager C, Lyon AR. Client progress monitoring and feedback in school-based mental health. Cognitive and Behavioral Practice. 2015;22(1):74-86.
Bohnenkamp JH, Glascoe T, Gracey KA, Epstein RA, Benningfield MM. Implementing clinical outcomes assessment in everyday school mental health practice. Child and Adolescent Psychiatric Clinics. 2015;24(2):399-413.
Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implementation Science. 2015;10(1):109.
Lyon AR, Cook CR, Locke J, Davis C, Powell BJ, Waltz TJ. Importance and feasibility of an adapted set of implementation strategies in schools. Journal of school psychology. 2019;76:66-77.
Awad M, Connors E. Promoting Measurement-Based Care in School Mental Health through Practice-Specific Supervision. Manuscript under review. 2022.
Baumann AA, Carothers BJ, Landsverk J, Kryzer E, Aarons GA, Brownson RC, et al. Evaluation of the implementation research institute: trainees’ publications and grant productivity. Administration and Policy in Mental Health and Mental Health Services Research. 2020;47(2):254-64.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PloS one. 2011;6(6):e20476.
Powell C. The Delphi technique: myths and realities. Journal of advanced nursing. 2003;41(4):376-82.
Lyon AR, Pullmann MD, Whitaker K, Ludwig K, Wasse JK, McCauley E. A Digital Feedback System to Support Implementation of Measurement-Based Care by School-Based Mental Health Clinicians. Journal of Clinical Child & Adolescent Psychology. 2019;48(sup1):S168-S79.
Lyon AR, Pullmann MD, Dorsey S, Martin P, Grigore AA, Becker EM, et al. Reliability, validity, and factor structure of the current assessment practice evaluation-revised (caper) in a national sample. The journal of behavioral health services & research. 2019;46(1):43-63.
Trochim W, Kane M. Concept mapping: an introduction to structured conceptualization in health care. International Journal for Quality in Health Care. 2005;17(3):187-91.
Herschell A, Kolko D, Baumann B, Davis AJCpr. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. 2010;30 4:448-66.
Valenstein-Mah H, Greer N, McKenzie L, Hansen L, Strom TQ, Wiltsey Stirman S, et al. Effectiveness of training methods for delivery of evidence-based psychotherapies: a systematic review. Implementation Science. 2020;15:1-17.
Rinad S. Beidas, Ph.D. ,, Julie M. Edmunds, M.A. ,, Steven C. Marcus, Ph.D. , and, Philip C. Kendall, Ph.D., A.B.P.P. Training and Consultation to Promote Implementation of an Empirically Supported Treatment: A Randomized Trial. Psychiatric Services. 2012;63(7):660-5.
Rones M, Hoagwood K. School-based mental health services: A research review. Clinical child and family psychology review. 2000;3(4):223-41.
Hoagwood KE, Cavaleri MA, Olin SS, Burns BJ, Slaton E, Gruttadaro D, et al. Family support in children’s mental health: A review and synthesis. Clinical child and family psychology review. 2010;13(1):1-45.
Weisz JR, Kuppens S, Eckshtain D, Ugueto AM, Hawley KM, Jensen-Doss A. Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA psychiatry. 2013;70(7):750-61.
Bruns EJ, Parker EM, Hensley S, Pullmann MD, Benjamin PH, Lyon AR, et al. The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare. Implementation Science. 2019;14(1):1-13.
Connors EH, Arora P, Curtis L, Stephan SH. Evidence-based assessment in school mental health. Cognitive and Behavioral Practice. 2015;22(1):60-73.
Richesson RL, Staes CJ, Douthit BJ, Thoureen T, Hatch DJ, Kawamoto K, et al. Measuring implementation feasibility of clinical decision support alerts for clinical practice recommendations. Journal of the American Medical Informatics Association. 2020;27(4):514-21.
Connors EH, Stephan SH, Lever N, Ereshefsky S, Mosby A, Bohnenkamp J. A National Initiative to Advance School Mental Health Performance Measurement in the US. Advances in School Mental Health Promotion. 2016;9(1):50-69.
Forman SG, Olin SS, Hoagwood KE, Crowe M, Saka N. Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health. 2009;1(1):26-36.

Table 1

Demographic and Professional Characteristics of Stakeholder Participants, N=52

Characteristic	Total sample	Providers	Researchers
	n %	n %	n %
Age
21-30 years	1 2	1 3	-- --
31-40 years	11 21	9 29	2 10
41-50 years	21 40	15 48	6 29
51-60 years	13 25	6 19	7 33
61 and over	6 12	-- --	6 29
Gender
Female	27 52	26 84	11 52
Male	25 48	5 16	10 48
Race/Ethnicity
Asian or Asian American	1 2	-- --	1 5
Caucasian or White	42 80	23 74	19 91
Hispanic (Spanish descent)	4 8	4 13	-- --
Latino/a/x (South or Central American descent)	2 3	2 7	-- --
Multiracial	3 6	2 7	1 5
Field of Training⁺
Clinical or Counseling Psychology	12 23	6 19	6 29
School Psychology	16 31	10 32	6 29
Social Work	11 21	10 32	1 5
Special Education	5 10	-- --	5 24
Multiple Fields	1 2	-- --	1 5
Professional Counseling	6 12	6 19	-- --
Substance Use/Addiction Counseling	1 2	1 3	-- --
Other (School Counseling, Rehabilitation Counseling Services)	11 21	9 29	2 10
Degree⁺
PhD	25 48	4 13	21 100
Ed.D	1 2	1 3	-- --
PsyD	1 2	1 3	-- --
LCSW	4 8	4 13	-- --
LGSW	1 2	1 3	-- --
LCPC	2 3	2 7	-- --
LMFT	1 2	1 3	-- --
BA/BS	5 10	5 16	-- --
MA/MS	15 29	15 48	-- --
Other (M.Ed, LMSW)	10 19	10 32	-- --
Urbanicity*⁺
Metro	38 72	20 65	18 86
Nonmetro	24 46	11 35	13 62
Note. N = 52 (n = 31 providers and n = 21 researchers). ⁺Participants selected all that apply for these characteristics so they add to greater than 100%
*Urbanicity for SMH researchers refers to their partnerships, not where they work personally

Table 2

Results of 33 Initial Implementation Strategies in Survey 1

#	Strategy	Importance		Feasibility		*Go-Zone	Survey 2 Decision
#	Strategy	M SD	Range	M SD	Range	*Go-Zone	Survey 2 Decision
14	Conduct ongoing training	4.39 0.84	2.0 - 5.0	3.61 0.95	2.0 - 5.0	1	Retained
27	Develop a detailed implementation plan or blueprint	4.34 0.80	2.0 - 5.0	3.80 0.97	1.0 - 5.0	1	Retained
9	Develop instruments to monitor and evaluate core components of the innovation/ new practice	4.27 0.81	2.0 - 5.0	3.51 0.87	1.0 - 5.0	1	Retained
2	Distribute educational materials	4.14 1.88	2.0 - 5.0	4.10 0.79	2.0 - 5.0	1	Collapsed with 1
16	Facilitation/problem-solving	4.29 0.91	1.0 - 5.0	3.76 0.90	2.0 - 5.0	1	Retained
5	Identify and prepare champions	4.30 0.74	3.0 - 5.0	3.60 0.93	2.0 - 5.0	1	Retained
29	Make implementation easier by removing burdensome documentation tasks	4.61 0.68	3.0 - 5.0	3.89 1.18	1.0 - 5.0	1	Retained
13	Make training dynamic	4.35 0.75	2.0 - 5.0	3.92 0.81	2.0 - 5.0	1	Retained
26	Promote adaptability	4.13 0.92	2.0 - 5.0	3.84 0.93	2.0 - 5.0	1	Retained
18	Provide ongoing consultation/coaching	4.38 0.85	1.0 - 5.0	3.66 1.07	1.0 - 5.0	1	Retained
1	Develop educational materials	3.96 0.83	2.0 - 5.0	3.84 0.91	1.0 - 5.0	2	Collapsed with 2
28	Identify early adopters	3.78 1.13	1.0 - 5.0	3.92 0.98	2.0 - 5.0	2	Retained
15	Use train the trainer strategies	3.85 1.04	1.0 - 5.0	3.60 0.88	2.0 - 5.0	2	Retained
23	Audit and provide feedback	3.96 0.93	2.0 - 5.0	3.28 0.77	2.0 - 5.0	3	Collapsed with 17 and 22
19	Build partnerships (i.e., coalitions) to support implementation	3.65 0.95	2.0 - 5.0	3.30 1.02	2.0 - 5.0	3	Collapsed with 21
32	Capture and share local knowledge	3.61 1.02	2.0 - 5.0	3.44 0.92	2.0 - 5.0	3	Removed
10	Change/alter environment	3.51 1.07	1.0 - 5.0	3.17 1.04	2.0 - 5.0	3	Removed
33	Conduct educational meetings with specific stakeholders	3.77 1.02	2.0 - 5.0	3.28 0.98	2.0 - 5.0	3	Removed
7	Conduct educational outreach visits	3.85 0.95	1.0 - 5.0	3.33 0.93	2.0 - 5.0	3	Removed
4	Conduct local needs assessment	3.89 1.10	1.0 - 5.0	3.48 0.86	2.0 - 5.0	3	Removed
21	Create a professional learning collaborative	3.76 0.97	1.0 - 5.0	3.22 0.99	2.0 - 5.0	3	Retained
8	Develop academic partnerships	3.44 1.00	1.0 - 5.0	3.15 0.90	2.0 - 5.0	3	Removed
11	Develop resource sharing agreements	3.58 1.20	1.0 - 5.0	3.19 1.01	2.0 - 5.0	3	Removed
6	Intervene/communicate with students, families, and other staff to enhance uptake and fidelity	3.89 1.06	1.0 - 5.0	3.15 0.94	2.0 - 5.0	3	Removed
20	Involve students, family members, and other staff	3.98 1.01	2.0 - 5.0	3.32 1.01	2.0 - 5.0	3	Retained
12	Model and simulate change	3.70 1.05	1.0 - 5.0	3.39 0.97	1.0 - 5.0	3	Removed
24	Obtain and use student and family feedback	3.92 0.88	2.0 - 5.0	3.39 1.03	2.0 - 5.0	3	Removed
30	Visit other sites	3.62 1.14	2.0 - 5.0	2.89 0.97	2.0 - 5.0	3	Removed
3	Assess for readiness and identify barriers and facilitators	4.31 0.82	2.0 - 5.0	3.47 0.96	2.0 - 5.0	4	Retained
22	Develop a quality monitoring system	4.38 0.74	3.0 - 5.0	3.36 0.92	2.0 - 5.0	4	Collapsed with 17 and 23
31	Facilitate relay of intervention and student data to school personnel	4.28 0.80	2.0 - 5.0	3.42 0.85	2.0 - 5.0	4	Removed
17	Monitor the progress of the implementation effort	4.41 0.69	3.0 - 5.0	3.43 0.94	2.0 - 5.0	4	Collapsed with 22 and 23
25	Provide practice-specific supervision	4.17 0.91	2.0 - 5.0	3.39 0.99	2.0 - 5.0	4	Retained

Note. All retained strategies appeared in Survey 2 with revised strategy names and/or definitions.

*Go-Zone 1 = High Feasibility/High Importance, Go-Zone 2 = High Feasibility/Low Importance, Go-Zone 3 = Low Feasibility/Low Importance, Go-Zone 4 = Low Feasibility/High Importance

Table 3.

Results of 21 Implementation Strategies in Survey 2

#	Strategy	Importance	Feasibility	Go-Zone
		M SD	M SD
1	Assess for readiness and identify barriers and facilitators	4.10 0.68	3.55 0.89	1
2	Identify and prepare champions	4.16 0.84	3.72 0.77	1
3	Develop a usable implementation plan	4.48 0.80	3.54 0.90	1
6	Offer a provider-informed menu of free, brief measures	4.40 0.79	4.06 1.02	1
7	Develop and provide access to training materials	4.37 0.84	3.85 0.92	1
21	Make implementation easier by removing burdensome documentation tasks	4.28 0.94	3.37 1.18	1
8	Make training dynamic	3.82 0.98	3.69 0.97	2
11	Use train the trainer strategies	3.61 1.13	3.39 1.04	2
12	Identify early adopters	3.67 0.87	3.56 0.91	2
15	Provide practice-specific supervision	3.98 1.00	3.40 1.11	2
4	Alter and provide individual- and system-level incentives	3.71 0.98	2.95 0.87	3
10	Support workflow adjustments	3.93 0.83	2.55 0.85	3
13	Facilitation	3.62 1.06	2.92 0.86	3
17	Involve students, family members, and other staff	4.05 0.91	2.98 1.09	3
18	Create a professional learning collaborative	3.64 0.97	3.18 1.06	3
5	Develop local policy that supports implementation	4.10 0.93	3.02 1.00	4
9	Conduct ongoing training*	4.07 0.83	3.30 0.91	4
14	Provide ongoing clinical consultation/coaching*	4.43 0.73	3.14 0.98	4
16	Monitor implementation progress and provide feedback*	4.30 0.70	3.30 0.95	4
19	Monitor fidelity to MBC core components*	4.24 0.69	3.31 0.81	4
20	Promote adaptability*	4.15 0.98	3.22 0.80	4

*These strategies were less than 0.50 of the mean cutoffs for feasibility, yet above the mean cutoff for importance.

Table 4.

Final List of 21 Implementation Strategies and Definitions for MBC in School Mental Health

Strategy

Definition

1. Assess for readiness and identify barriers and facilitators

Assess readiness for MBC at provider, administration, and school setting levels. Identify strengths or facilitators that can support the implementation effort and barriers that might get in the way. This could occur before, during, and/or after implementation with providers, primarily, as well as other stakeholder groups.

2. Identify and prepare champions

Identify individuals who are passionate about MBC in schools and are influencers or informal leaders among fellow providers. Prepare and support them to facilitate implementation, support their peers, and overcome or address indifference or resistance that MBC may provoke in a school or district. There may be more than one champion per school site. Sites may have the option to adjust this title to local language (e.g., MBC key opinion leader, lead provider, coach, intervention specialist).

3. Develop a usable implementation plan

Develop a usable plan for implementation built around student outcomes as the ultimate goal of MBC implementation effort. The implementation plan will detail processes and strategies that will be used to achieve those goals. The plan should also include timeframe and milestones, roles and responsibilities of all stakeholders, and appropriate performance/progress measures. Use and update this plan to guide the implementation effort over time. It will be used to promote excitement and buy-in, collaborative planning, clear communication and training and adaptive implementation over time.

4. Alter and provide individual- and system-level incentives

Work to provide individual-level (e.g., provider recognition, acknowledgement, gift card) and system-level (e.g., grant money, free training and consultative support) incentives to districts or schools to participate and engage in an MBC implementation effort.

5. Develop local policy that supports implementation

Develop local school system policy that establishes rules, expectations, and guidelines for MBC implementation.

6. Offer a provider-informed menu of free, brief measures

Engage providers in a discussion about measure selection to select and distribute a small number of progress monitoring tools. Emphasize tools that are free, brief, and easy to score.

7. Develop and provide access to training materials

Training materials (i.e., a curriculum, toolkit, or guide) for MH professionals would include what MBC is, why MBC is important, goals of MBC, clear steps to follow, examples and non-examples of proper MBC, implementation scripts, practice profiles, timelines, and rating scales for use. The study team would develop these materials with school provider stakeholder input and would work with schools and mental health agencies to distribute materials to school providers electronically. Materials would be made available to providers following the training.

8. Make training dynamic

Ensure the initial training is interactive, experiential and relevant to providers (e.g., to include role plays, MBC practice examples and non-examples, MBC research base, planning ahead for MBC implementation with students served, and discussion). Make information available in multiple formats. Vary how information is delivered for various professional development schedules and structures.

9. Conduct ongoing training

Plan for and conduct one or more “booster” or follow-up trainings after the initial training. (Note: This is different from ongoing clinical consultation/coaching, or supervision.)

10. Support workflow adjustments

Provide protected time for individualized implementation planning about how the provider can integrate MBC into their existing workflow and problem solve anticipated barriers. This is intended to acknowledge providers’ limited time and provide support for self-reflection and personalized action planning. May involve engaging providers’ supervisor or building administrator for support. This could occur at the initial training, booster training, or during ongoing consultation.

11. Use train the trainer strategies

Train designated, local school providers to train other school mental health providers in MBC using a systematic process to support ongoing implementation and sustainability.

12. Identify early adopters

Identify early adopters (i.e., individuals who are particularly open to change) of MBC within the school, district or community agency to learn from their experiences and demonstrate the benefit of MBC to other providers. Early adopters could share their experiences of success with others and be involved in ongoing training and consultation efforts if they are willing. This strategy is used after implementation has started for everyone; it is different than piloting with a small group of enthusiastic providers first before implementing.

13. Facilitation

A process of interactive implementation support that is provided by an internal or external facilitator to the whole school or district system. Facilitation should be non-evaluative, informative and part of a supportive interpersonal relationship. Usually provided by someone who works with school leaders, providers, and all other stakeholders to problem solve and tailor the types of support provided based on specific barriers or challenges with MBC implementation. For example, a facilitator could help address systemic barriers to implementation based on what stakeholders report and recommend. This is different from ongoing clinical coaching or consultation to providers to help them implement MBC with their students directly.

14. Provide ongoing clinical consultation/coaching

Provide ongoing clinical consultation and coaching by one or more experts or trained clinical peers. Consultation and coaching would be non-evaluative, flexible, individualized, and focused on helping providers improve their MBC implementation. This includes problem solving and performance feedback throughout implementation. (NOTE: Clinical consultation / coaching is typically differentiated from the next strategy, supervision, which is usually provided by an internal individual who has supervisory authority over the implementer.)

15. Provide practice-specific supervision

Provide school providers with supervision focusing on MBC. Supervisors are in a position of authority and support school providers who deliver new practices with evaluative feedback via performance assessment. (Note: Supervision is typically differentiated from consultation/ coaching, which may be provided by an internal or external individual who may or may not have authority over the implementer.)

16. Monitor implementation progress and provide feedback

Collect and summarize data regarding MBC implementation outcomes (fidelity, acceptability, how many providers are using it) over a specified time period and give it to administrators, school personnel and/or providers to monitor, evaluate, and support providers’ MBC practices. The purpose of this strategy is to continuously improve the quality of implementation and inform data-driven, real-time decisions about what supports providers need most. To do this, a quality monitoring system and procedures would first need to be developed. Also referred to as “audit and provide feedback”. (Note: This is included in strategies 13. Facilitation and 14. Ongoing clinical consultation/coaching.)

17. Involve students, family members, and other staff

Ask students, family members, and providers, as those receiving and providing MBC, to provide input and recommendations about implementation to improve practice and quality. Topics may include how school providers can most effectively implement MBC (to collect, share, and act on student progress data), how to ensure students and families can be actively involved in MBC, ways to make MBC purpose clear to everyone, ensuring an equal student-parent-provider partnership, addressing concerns or barriers, and/or what implementation supports are needed.

18. Create a professional learning collaborative

Facilitate the formation of school provider groups within or between school systems or mental health agencies to foster a collaborative learning environment to improve MBC implementation. Providers could network with others in their district or beyond who are also implementing MBC to share resources, lessons learned, and support with the help of a learning collaborative facilitator. The learning community would be organized, developed and coordinated by a research team or implementation consultant (not the providers).

19. Monitor fidelity to MBC core components

Integrate measurement strategies to assess the degree to which MBC core components (i.e., collecting, sharing, and acting on the student progress data) is occurring during implementation. For example, study team or clinical supervisors could review IEPs, 504 plans or treatment plans for documentation of MBC. The purpose is to inform ongoing quality improvement and implementation supports.

20. Promote adaptability

Identify ways MBC can be tailored or adapted to best fit with the school/classroom context, meet local needs (e.g., selecting measures most appropriate for student characteristics, cultural and linguistic competencies) and clarify which elements of MBC must be maintained to preserve fidelity. The MBC implementation study team and school personnel (mental health providers, administrators) would work together on adaptations or tailoring needed to improve feasibility, acceptability, and appropriateness of MBC. Adaptations should be documented and based on provider, student and/or family feedback after initial implementation with fidelity.

21. Make implementation easier by removing burdensome documentation tasks

Remove or alleviate burdensome tasks or documentation that could come with implementing MBC (e.g., removing unnecessary or unused data forms, streamlining duplicative paperwork, require only minimal necessary documentation, and make sure all data collected are used). This should apply to measures collected (i.e., improve efficiency with user-friendly forms and auto scoring) and progress note documentation (i.e., templates to document MBC data results and how they were used in session with the student and family).

Survey1MBCDefinitionsandVignettes.pdf

Download PDF

Reviewer #2 agreed at journal
20 Feb, 2022
Reviews received at journal
18 Feb, 2022
Reviewers invited by journal
18 Feb, 2022
Reviewer #1 agreed at journal
17 Feb, 2022
Submission checks completed at journal
10 Feb, 2022
Editor invited by journal
10 Feb, 2022
Editor assigned by journal
10 Feb, 2022
First submitted to journal
09 Feb, 2022

You are reading this latest preprint version

Implementation Strategies to Promote Measurement-based Care in Schools: Evidence from Mental Health Experts Across the United States

Status:

Version 2

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Survey 1 Strategy Ratings

Survey 2 Strategy Ratings

Survey 2 Top-Rated Strategies

Stakeholder Group Comparisons

Discussion

Limitations

Conclusion And Future Directions

Abbreviations

Declarations

References

Tables

Supplementary Files

Status:

Version 2