How Low Can You Go? Examining the Effects of Brief Online Training and Post-Training Consultation Dose on Implementation Mechanisms and Outcomes for Measurement-Based Care

doi:10.21203/rs.3.rs-592947/v1

Download PDF

Research

How Low Can You Go? Examining the Effects of Brief Online Training and Post-Training Consultation Dose on Implementation Mechanisms and Outcomes for Measurement-Based Care

https://doi.org/10.21203/rs.3.rs-592947/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 21 Jul, 2022

Read the published version in Implementation Science Communications →

Version 1

posted

You are reading this latest preprint version

Background: Initial training and ongoing post-training consultation are among the most common implementation strategies used to change clinician practice. However, extant research has not experimentally investigated the optimal dosages of consultation necessary to produce desired outcomes. Moreover, the degree to which training and consultation engage theoretical implementation mechanisms – such as provider knowledge, skills, and attitudes – is not well understood. This study examined the effects of a brief online training and varying dosages of post-training consultation (BOLT+PTC) on implementation mechanisms and outcomes for measurement-based care (MBC) practices delivered in education sector mental health services.

Methods: A national sample of 75 clinicians who provide mental health interventions to children and adolescents in schools were randomly assigned to BOLT+PTC or control (services as usual). Those in BOLT+PTC were further randomized to 2-, 4-, or 8- week consultation conditions. Self-reported MBC knowledge, skills, attitudes, and use (including standardized assessment, individualized assessment, and assessment-informed treatment modification) were collected for 32 weeks. Multilevel models were used to examine main effects of BOLT+PTC versus control on MBC use at the end of consultation and over time, as well as comparisons among PTC dosage conditions and theorized mechanisms (skills, attitudes, knowledge).

Results: There was a significant linear effect of BOLT+PTC over time on standardized assessment use (b = .02, p < .01), and a significant quadratic effect of BOLT+PTC over time on individualized assessment use (b = .04, p < .001), but no significant effect on treatment modification. BOLT + any level of PTC resulted in higher MBC knowledge and larger growth in MBC skill over the intervention period as compared to control. PTC dosage levels were inconsistently predictive of outcomes, providing no clear evidence for added benefit of higher PTC dosage.

Conclusions: Online training and consultation in MBC had effects on standardized and individualized assessment use among clinicians as compared to services as usual with no consistent benefit detected for increased consultation dosage. Continued research investigating optimal dosages and mechanisms of these established implementation strategies is needed to ensure training and consultation resources are deployed efficiently to impact clinician practices.

Clinical Trial Registration: Not applicable.

Psychiatry

implementation strategy optimization

measurement-based care

education sector mental health

mental health treatment in schools

consultation

Initial training and post-training consultation are cornerstone implementation strategies, but their mechanisms of action are rarely explicitly assessed, and little is known about the effects of consultation dosage.
This study is innovative in that it evaluated implementation mechanisms for training and consultation to support the use of measurement-based care in education sector mental health services, and is the first to experimentally manipulate consultation dosage.
Results suggest that the implementation strategies were effective in changing MBC practices among clinicians, but little evidence was found for a consistent benefit of higher dosages of consultation. This may indicate new, unexplored efficiencies surrounding the implementation of evidence-based practices in mental health.

Over the past two decades, studies have repeatedly explored the extent to which initial training must be supplemented with additional implementation supports to effect meaningful changes in professional practice. Many reviews indicate that post-training consultation or support is necessary, but the strength of the evidence remains mixed [1–4]. Recent literature points to the limitations of “workshop only” trainings, and suggested that they are likely to have a greater impact on the adherence and competence of practitioners when augmented with practice-specific implementation supports and/or ongoing consultation to promote workplace-specific adaptations and skill application [5–7]. As such, initial training and post-training consultation are two cornerstone – and complementary – implementation strategies to support the implementation of evidence-based practices [8, 9]. This is especially true in mental health care, where most evidence-based practices are complex psychosocial processes that require significant professional development support to implement effectively [10].

Implementation Strategies: Training and Consultation

Among the numerous implementation strategies that have been identified to potentially promote adoption, delivery, and sustainment of interventions [11,12], initial training and post-training consultation have often proven effective for evidence-based mental health practice implementation [3,8,9]. Initial training generally refers to a combination of didactic content covering intervention materials or protocols and active or experiential learning such as in vivo demonstration or role plays to apply skills and receive feedback [1]. Ongoing, post-training consultation refers to practice-specific coaching by one or more experts in the intervention strategy [13]. Consultation is different from supervision in that the consultant may be internal or external to the organization and does not have direct authority over the implementer [14]. Moreover, supervision can be administrative in nature [15] and consultation specifically refers to case-specific skill application and refinement of an evidence-based clinical practice [16,17]. Together, initial training and post-training consultation are complementary and foundational implementation strategies.

Training is the starting point for many professional behavior change efforts, and is associated with initial, post-training improvements in provider attitudes, knowledge, and skill acquisition [4]. Increasingly, training in mental health is provided online, something that has only intensified following the emergence of the COVID-19 pandemic [18]. Fortunately, online training is at least as effective as in-person approaches [19-22], eliminates training-related travel time, reduces costs, and allows for self-paced administration of content, thus improving accessibility and feasibility [20,23].

Importantly, initial training likely necessitates the addition of post-training consultation or support to effect changes in professional practice [1-4,24,25]. However, the optimal duration or frequency of ongoing supports following initial training are not well known. In fact, the most recent review on this topic found that post-training consultation was variably effective in improving uptake [4]. Inconsistencies in these findings suggest that it is important to better understand the processes through which consultation improves implementation outcomes.

Consultation dose

Although post-training consultation is a promising implementation strategy, very little is known about the optimal frequency, intensity, or duration of post-training consultation to ensure its impact. In fact, a recent review of randomized controlled trials concluded that variations in consultation dose, intensity and frequency across studies precluded their ability to conclusively identify the effect of consultation practices, for which additional research is needed [4]. Efficiency is important given that post-training consultation can easily increase the cost of training by 50% or more [26]. An uncontrolled study found that higher consultation doses were associated with greater changes in knowledge and attitudes surrounding an evidence-based intervention protocol for youth anxiety [27]. Although the moderating effects of consultation dose are typically examined observationally or quasi-experimentally [23], the current study explores consultation dose by manipulating it experimentally.

Implementation Mechanisms: Provider Knowledge, Skill, and Attitudes

Despite more precise identification of implementation strategies to address the large and persistent gaps between typical and optimal practice in contemporary healthcare [12], there is little information available about how implementation strategies exert their influence on implementation outcomes. To ensure implementation strategies accurately match barriers and promote parsimonious and efficient change efforts, current research has focused on the identification of implementation mechanisms [28]. Implementation mechanisms are the processes or events through which an implementation strategy operates to affect desired implementation outcomes [29]. Unfortunately, few studies have experimentally evaluated implementation mechanisms. In mental and behavioral health, for instance, two systematic reviews have recently underscored how few studies test implementation mechanisms of change [29,30]. Whether in schools or other healthcare delivery sectors, there is growing recognition that implementation strategy optimization will be greatly facilitated by clear identification and testing of the mechanisms through which implementation outcomes (e.g., adoption) are improved [28,31,32].

Despite the ubiquity of training and consultation in the implementation of evidence-based mental health service delivery, few studies have articulated or evaluated their mechanisms. Theorized mechanisms for training and consultation include provider changes in (1) knowledge, (2) attitudes, and (3) acquisition of skills. Knowledge and attitudes surrounding new practices are among the most frequently-identified candidate mechanisms for training efforts [33]. Indeed, previous studies have shown that training often improves EBP knowledge, and, in turn, more knowledgeable and competent clinicians have generally been found to exhibit superior training and consultation outcomes [33-35]. Positive attitudes are often predictive of provider adoption, engagement in training and consultation, adherence, and skill [36-39]. Further, training has been found to enhance initial skill acquisition [4,40] and post-training consultation subsequently reinforces these gains and helps to maintain clinician skills [41]. In one study of community mental health clinicians’ acquisition of evidence-based assess practices following a training in a flexible psychotherapy protocol, attitudes improved following training alone, but skill continued to improve over the course of consultation [42]. Other studies have found similar evidence for the value of consultation for maintaining and promoting these changes [24].

Measurement-Based Care in Education Sector Mental Health

The current study focuses on understanding the impact of training and consultation strategies for measurement-based care (MBC) – an evidence-based practice that involves the ongoing use of progress and outcome data to adjust treatment [43,44] – on implementation mechanisms and outcomes within mental health services delivered in the education sector. MBC involves both standardized and individualized assessment measures [45-47] as well as a focus on reviewing progress data with the patient to inform collaborative decisions about individualized treatment adjustments [48,49].

MBC is particularly suited for treatment delivered in schools, where it is perceived by clinicians and students to be feasible, acceptable, and to facilitate progress toward treatment goals [50-52]. Research over the past 25 years has consistently found that schools are the most common service setting for the delivery of child and adolescent mental healthcare [53,54]. Yet, use of evidence-based mental health treatment in school settings – including MBC – has lagged behind services in other sectors [55]. School clinicians rarely receive pre-service (e.g., during graduate training) or in-service professional development in MBC and, as a result, rates of use are low [50,56].

The implementation of MBC in the education sector is consistent with increasing calls for initiatives that enhance the quality of services among settings, clinicians, and service recipients who would not otherwise have access to evidence-based care [57,58]. However, while pragmatic implementation approaches are greatly needed to support MBC across service sectors, few established implementation strategies exist [48] and even fewer studies have evaluated strategies for improving use of MBC practices among school clinicians. Existing training and consultation strategies used to implement MBC practices in schools have resulted in improvements in knowledge, attitudes and use of practices initially and over time [42,59], pointing to them as potential mechanisms for MBC implementation strategy effects.

Current Study

The current study was designed to better understand the impact of a brief online training (BOLT) and post-training consultation (PTC) on implementation mechanisms and outcomes. Moreover, our design facilitated examination of the effects of PTC dosage levels on outcomes. This was achieved through random assignment of mental health providers to BOLT+PTC versus a services-as-usual control condition. Within the BOLT+PTC condition, providers were further randomized to 2- , 4- , or 8- week consultation conditions. This study focused on increasing school mental health providers’ use of standardized assessments, individualized assessments, and assessment-informed treatment modifications (all components of MBC practice) [44]. Accordingly, the current study examined three specific aims by testing: (1) the impact BOLT+PTC supports on MBC practices; (2) differential effects of a 2-, 4- or 8- week dosage of consultation on MBC practices; and (3) implementation mechanisms such as MBC knowledge, skill, and attitudes (i.e., perceived benefit and perceived harm) that are hypothesized to activate favorable MBC implementation outcomes.

Participants

Participants included a national sample of N = 75 clinicians who provide mental health interventions to students on their school campus. Participants were primarily Female (n = 68, 91%) and White (n = 55, 73%), working in elementary, middle and high schools. The most common role was mental health counselor, followed by school social worker, school psychologist, school counselor, and other professional roles. Demographic and professional characteristics for participating clinicians are presented in Table 1.

Procedure

Recruitment

Participants were recruited via numerous professional networks, listservs and social media. Inclusion criteria were minimal to enhance generalizability and only included the requirements that 1) participants routinely provided individual-level mental health interventions or therapy and 2) spent ≥50% of their time providing services in schools. This was done to help ensure the representativeness of the sample compared to the clinicians who would ultimately access the training and consultation supports when later disseminated at a large scale. The study team conducted informed consent meetings via phone with prospective participants and received written/verbal consent consistent with procedures approved by the University of Washington institutional review board. Recruitment lasted for a period of approximately six weeks to achieve the desired sample size.

Randomization

All participants were randomly assigned to either a BOLT+PTC condition or a service as usual control condition (see below). Clinicians in the BOLT+PTC condition (n = 37) were further randomized to 2 (n = 14), 4 (n = 10), or 8 (n = 13) weeks of consultation. Clinicians in the control condition only completed study measures while continuing to provide services as usual. See Additional File 1 for the study CONSORT diagram.

Data collection

All data were collected via online surveys and self-reported by participants. After enrolling in the study, all participants completed pre-training measures of their demographic, professional, and caseload characteristics and MBC knowledge, skill, attitudes, and use. These measures were collected weekly for 32 weeks following study enrollment. All participants received incentives for the time spent completing assessments throughout the study.

Training and consultation

The online training and post-training consultation strategies were developed via an iterative human-centered design process intended to ensure their efficiency and contextual fit [60,61].

Online training. After completing pre-training measures, participants assigned to any BOLT+PTC condition were asked to complete the online training within 2 weeks. Training included a series of interactive modules addressing the following content: (1) Utility of MBC in school mental health; (2) Administration and interpretation of measures; (3) Delivery of collaborative feedback; (4) Treatment goal-setting and prioritization of monitoring targets; (5) Selecting and using standardized assessments; (6) Selecting and using individualized assessments, (7) Assessment-driven clinical decision-making; and (8) Strategies to support continued use. The interactive online training modules are accompanied by a variety of support materials (e.g., tools to help integrate MBC into clinicians’ workflow; job-aids for introducing assessments and providing feedback on assessment results; a commonly used youth measures reference guide) via an online learning management system.

Consultation. The consultation model consisted of (1) one-hour small group (3-5 clinicians) live consultation sessions led by an expert MBC consultant and (2) asynchronous, message board discussions (hosted on the learning management system and moderated by the same consultant). Regardless of the consultation dosage, live consultation calls followed a standard sequence: (a) introduction and orientation to the session; (b) brief case presentations including MBC strategies used; (c) group discussion of appropriate next MBC steps for the case, including discussion of alternative therapeutic approaches/ strategies if MBC indicates that a change in treatment target or intervention strategy is needed; (d) expert consultant recommendations (as appropriate); and (e) wrap up, homework assignments, and additional resources. Asynchronous message board discussion provided a central location where clinicians reported on their experiences completing homework. All participants in the BOLT+PTC condition (n = 37) posted more than once on the discussion boards.

Services as usual

Typical education sector mental health services tend to include a diverse array of assessment strategies that may include some inconsistent use of formal assessment and monitoring measures [50,56]. Clinicians in this condition only completed study assessments.

Measures

Clinician demographic, professional, and caseload characteristics

Clinician demographic, professional and caseload characteristics were collected using a self-reported questionnaire developed by the study team, informed by those used in prior school-based implementation research (e.g., [62]). Participants completed this questionnaire upon study enrollment.

MBC Knowledge Questionnaire (MBCKQ)

Modeled on the Knowledge of Evidence-Based Services Questionnaire [63], the MBCKQ was designed to assess factual and procedural knowledge about MBC. The 28-item, multiple-choice MBCKQ was iteratively developed based on the key content and learning objectives of the MBC training modules and administered at baseline and 2, 4, 6, 8, 10, 16, 20, 24, 28, and 32 weeks.

MBC skill

Clinicians responded to 10 Likert-style items assessing MBC skills including selection and administration of measures, progress monitoring, treatment integration/modification based on the results, and feedback to clients. Responses range from 1 (‘‘Minimal’’) to 5 (‘‘Advanced’’). This scale has previously demonstrated good internal consistency (α = .85) when used with school-based clinicians [50]. In the current sample, α = .90. It was also administered at baseline and subsequently at 2, 4, 6, 8, 10, 16, 20, 24, 28, and 32 weeks.

MBC attitudes

The Monitoring and Feedback Attitudes Scale (MFA) [64] was used to assess clinician attitudes toward ongoing assessment of mental health problems and the provision of client feedback (e.g., “negative feedback to clients would decrease motivation/engagement in treatment”). Responses range from 1 (“Strongly Disagree”) to 5 (“Strongly Agree”). The MFA has two subscales: (1) Benefit (i.e., facilitating collaboration with clients) and (2) Harm (i.e., harmful for therapeutic alliance, misuse by administrators). In the current sample, the MFA subscales demonstrated strong internal reliability (α = .91 and .88, respectively) and was administered at baseline and 2, 4, 6, 8, 10, 16, 20, 24, 28, and 32 weeks.

MBC practices

Clinician self-reported use of MBC practices was measured by their completion of the Current Assessment Practice Evaluation – Revised (CAPER) [46], a measure of MBC penetration. The CAPER is a 7-item self-report instrument that allows clinicians to self-report their use of assessments in their clinical practice during the previous month and previous week. Clinicians indicate the percentage of their caseload with whom they have engaged in each of the seven assessment activities. Response options for each activity are on a 4-point scale (1 = “None [0%],” 2 = “Some [1-39%],” 3 = “Half [40-60%],” 4 = “Most [61-100%]”). The CAPER has three subscales, which are (1) Standardized assessments (e.g., % of case load administered standardized assessment during the last week); (2) Individualized assessments (e.g., % of caseload systematically tracked individualized outcomes last week); and, (3) Treatment modification (e.g., % of clients whose overall treatment plan altered based on assessment data during the last week). Previous versions have been found to be sensitive to training [42]. The CAPER demonstrated good internal consistency across its three subscales in the current study (α = .85, .94, .87). Clinicians completed the CAPER every week for 32 weeks, including baseline; total CAPER scores for each subscale were used as implementation outcomes.

Analyses

The main goal of the current study was to test whether the BOLT+PTC strategies led to improvements in MBC implementation mechanisms and outcomes, relative to a no-training services as usual condition. Specific MBC practices we measured using the CAPER were standardized and individualized assessment use and treatment modification informed by assessment data collected. Our second goal was to test whether the dose of PTC was differentially related to implementation outcomes (i.e., MBC practices). Finally, we tested the main effects of training and consultation dose on consultation mechanisms (i.e., MBC knowledge, attitudes, and skill).

We used R [65] for all analyses, and tested our main hypotheses using multilevel models (MLM) with the statistical package ‘nlme’ [66]. MLMs allow for the analysis of clustered data, such as multiple observations of clinicians collected over time, and allows for missing data at the observation level. Because there were 32 weeks of CAPER data, we were able to compare multiple polynomial forms of change over time. We centered time such that the intercept reflected the end of consultation, regardless of consultation dosage. For instance, for those with 2 weeks of PTC the intercept was centered at Week 4 and for those with 4 weeks of PTC, the intercept was centered at Week 6. This means that the main effects of intervention reflect differences across groups in levels of the outcomes at the end of consultation, or after the first two weeks of observations for the no-consultation control group.

We compared polynomials (quadratic and cubic effects) to a piecewise model, which estimated separate models of change during the consultation period and change in the post-consultation period. Models were chosen based on fit; we selected a given model as “better” fitting when all indices (e.g., BIC, AIC and -2LL test) agreed to avoid capitalizing on chance. Because we had a relatively small sample size, we only estimated random intercepts and linear slopes; estimating random quadratic slopes produced model convergence problems.

We used 3 orthogonal contrast codes to compare the effects of intervention across conditions. The first compared the effects of BOLT+PTC to control. The second compared the effects of receiving PTC for 4 or 8 weeks to receiving PTC for 2 weeks, and the final contrast code compared receiving 4 vs. 8 weeks of PTC.

Descriptive Statistics

Table 1 presents overall descriptive statistics for the main sample for the current study. Table 2 presents means and SDs for implementation outcomes and mechanisms across time.

Main Effects on Implementation Outcomes

First, we examined whether receiving BOLT+PTC produced changes in MBC practices during the post-consultation period (weeks 8 – 32), relative to the control group, and without accounting for change over time. Generally, mean differences were moderate for use of standardized assessment (Cohen’s d = .37, p = .11) and individualized assessment (Cohen’s d = .49, p = .04), but close to zero for treatment modification (Cohen’s d = .05, p = .84). However, it is important to note that these main effect estimates were non-significant or trending towards non-significance.

Change over Time in Outcomes

Next, we modeled change over time in the main outcomes, MBC practices. For standardized assessment, a quadratic term exhibited best fit to the data indicating a curvilinear pattern of change (see Figure 1), but a linear only model fit the data for individualized assessment and treatment modification.

Main effects over time

We tested whether receiving BOLT+PTC led to post-training changes in MBC practices over time. Table 3 presents these findings. Results indicated relatively similar findings across outcomes, in that those clinicians who received training and any consultation exhibited better outcomes over time, while those in the control group exhibited declines in MBC practices over time, with those declines generally slowing with time. Figure 1 also illustrates the effects of BOLT+PTC over time on MBC practices. Specifically, for both standardized assessment (b = .02, p = <.01) and individualized assessment (b = .04, p < .001), there was a significant intervention effect on change over time, such that participants in the BOLT+PTC condition exhibited less decline relative to those in the control condition. While clinicians in BOLT+PTC essentially maintained their use of standardized and individualized assessments, engagement with these MBC practices in the control group declined over time. On the other hand, we observed no effects of BOLT+PTC on treatment modification.

Effects of PTC dose on main outcomes over time

We next explored whether PTC dose influenced outcomes, by replicating the analyses above with additional contrast codes for different PTC doses. Across models, there was little evidence of a consistent pattern of findings. Although some effects reached the traditional significance level, there was not consistent or strong evidence that any one PTC dose condition had a stronger impact on outcomes than any other. Because of the small sample size of the individual training groups, and the inconsistent effects, we refrain from interpreting them as evidence of a clear dose effect of PTC. See Table 4.

Effects on Implementation Mechanisms

Next, we tested whether BOLT+PTC was associated with change over time in the theoretical implementation mechanisms (MBC knowledge, attitudes, and skill). We used a similar approach to the analyses above, using multilevel models with time centered at the end of consultation for each group. At the end of consultation, participants in the BOLT+PTC conditions reported higher levels of MBC knowledge (b = .06, p = .019), but not higher levels of MBC skill (b = .13, p = .33), or more positive MBC attitudes including perceived benefit (b = .16, p = .06) or perceived harm (b = -.12, p = .34) compared to control group participants. Participants in the BOLT+PTC condition exhibited larger increases across the consultation period in MBC skills (b = .03, p <.001), although these increases slowed over time (b = -0.00091, p = <.001). We observed no other effects of BOLT+PTC on rates of change over time.

We also explored whether PTC dose was associated with differential outcomes for theorized mechanisms (skill, attitudes, and knowledge), by replicating the analyses above with additional contrast codes for different PTC doses. Across all outcomes, there was no evidence that PTC dose was associated with differences in mechanisms.

This study tested the impact of an efficient, brief online training and post-training consultation program for MBC with school-based mental health clinicians. In addition to examining the effects of BOLT + PTC, it was the first study to experimentally manipulate PTC dose to evaluate the amount of support that may be needed to promote effective implementation. Consistent with current implementation research design guidelines to advance the precision of implementation science results, we examined the impact of BOLT and various dosages of BOLT + PTC on implementation outcomes (e.g., MBC practices) as well as implementation mechanisms (i.e., MBC knowledge, attitudes and skill).

In terms of implementation outcomes, effects were apparent over time for the impact of BOLT + PTC on both standardized and individualized assessment use, but not for treatment modification, even though each of these was an explicit focus of the training and consultation. Consistent with this, the developers of the CAPER have reported the lowest mean ratings for the treatment modification subscale [46]. This is also consistent with prior findings that have found less change, and less sustainment of changes, in treatment modifications following training and consultation in MBC practices [42]. It may be the case that changing one’s practice to collect standardized or individualized measures is more straightforward than using those measures to inform treatment modifications. Indeed, determining changes to treatment plans is often more complicated than instrument administration alone. Studies have pointed to the importance of ensuring clinical decision support during MBC implementation [67], which some measurement feedback systems and ongoing clinical consultation can provide. Although MBC has been associated with improved outcomes among a wide variety of presenting concerns and treatment modalities [43, 44], it is possible that using MBC for treatment planning is more ambiguous when implemented without a specific evidence-based intervention. Treatment modification may be easier if MBC is implemented alongside a broader practice change initiative, such as those that include transdiagnostic or common elements of evidence-based practices [68–70] or even clear and consistent specification of usual care treatment elements [71]. The current sample of clinicians delivered MBC-enhanced “usual care” mental health services in schools, rather than a specific evidence-based intervention.

Notably, these clinicians also delivered usual care services in the educational setting, where there is limited research on evidence-based practice implementation [55]. Given the importance of better understanding setting-specific rates of change, deterioration, outcomes, and treatment interventions that may influence MBC implementation and effectiveness [72], additional research examining how MBC practices can more effectively inform treatment modifications in school mental health treatment – and beyond – is greatly needed.

Regarding implementation mechanisms, two of the three theorized mechanisms were impacted for the BOLT + PCT group, relative to controls. This included higher levels of MBC knowledge immediately following consultation and larger gains in MBC skills across the consultation period, but no observed changes in MBC attitudes. As noted above, EBP attitudes have been inconsistently associated with implementation outcomes across service sectors and interventions. Our findings suggest that they might be deprioritized as candidate mechanisms for training and consultation, related to changes in knowledge and skills.

Research is increasingly investigating “how low can you go” with regard to implementation processes and pursuing pragmatic and cost-effective implementation supports [36, 73, 74]. Related, our findings regarding consultation dose were not as anticipated. Specifically, we observed very small differences among the different PTC groups (2, 4, or 8 weeks), indicating that while post-training consultation is critical, higher doses may not have improved its impact in this sample. However, many questions remain surrounding optimal consultation dosage, and replication with larger samples of clinicians is needed. Aside from aforementioned between-group power limitations, there are several possible explanations for our findings. First, it may be that MBC practices are simpler, relative to evidence-based treatment protocols which often use training and PTC to support clinicians’ adoption and ongoing implementation with fidelity. Therefore, perhaps less consultation is more likely to be adequate for MBC than for manualized, evidence-based interventions. Indeed, it has been suggested that less complex interventions may need fewer implementation supports to be successfully adopted [75]. On the other hand, our findings that standardized and individualized assessment administration changed more over time as compared to treatment modifications suggests that treatment modification could be a more complex or challenging practice to change than just collecting new measures. It is also possible that none of the consultation conditions were sufficient to effect this change or to support clinicians in determining what other intervention changes were indicated when measurement suggests a lack of progress. In BOLT + PTC, treatment modifications in response to assessment data were discussed more generally to enhance usual care, rather than in adherence to a particular set of intervention practices or expectation to deliver a manualized intervention. Future work should explore whether the brief model could be augmented to focus more explicitly on understanding when or how to adjust care. In this way and others, MBC – and MBC-facilitated treatment more generally – may continue to be a practice that is “simple, but not easy.”

Limitations

Findings of the current study must be interpreted within the context of several limitations. First, we were unable to include observational measures of MBC practices in this initial study. Although self-reported clinical practices are typical in implementation research for practicality and resource constraints particularly of pilot trials, observational measures would have been more robust and are a recommended future direction for related work. Second, based on previously-described research documenting the limitations of workshop training alone, we opted not to include a training-only condition to examine the differential effects of training compared to training plus consultation. Third, collection of student clinical outcomes was outside the scope of this study due to its pilot nature and our explicit focus on the most proximal implementation mechanisms and outcomes linked to the implementation supports provided. Finally, as indicated above, the sample sizes of each PTC dosage condition limited the conclusions that could be drawn about the differential effects of a greater number of weeks of consultation.

Results from this study indicate that efficient training and consultation supports can produce meaningful practice change surrounding the administration of assessments. However, better understanding the optimal dosage of consultation when paired with high-quality, active training requires further investigation, particularly for MBC delivered in schools. This is in line with recent evidence that even a condition without any post-training consultation – but with other potentially low-cost (but high quality) supports like handouts, video tutorials, and access to online supports – was able to yield improvements in some aspects of fidelity, including those that might be most important for patient outcomes [7]. This points to the necessity of continuing to identify and test the mechanisms engaged by various implementation strategies, especially training and consultation processes, to determine the most parsimonious and efficient approaches. Certainly, the incremental impact of consultation relative to alternative or adjunctive post-training support strategies, such as measurement feedback systems [76, 77], should be further explored as research on successful MBC implementation strategies unfolds.

BOLT: Brief Online Training

CAPER: Current Assessment Practice Evaluation – Revised

MBC: Measurement-Based Care

MBCKQ: Measurement-Based Care Knowledge Questionnaire

MFA: Monitoring and Feedback Attitudes Scale

MLM: Multilevel Model

PTC: Post-Training Consultation

Ethics approval and consent to participate

All study procedures were reviewed and approved by the University of Washington institutional review board. The study team conducted informed consent meetings via phone with prospective participants and received written/verbal consent consistent with IRB-approved procedure.

Consent for publication

Not applicable.

Availability of data and material

The datasets generated during and/or analyzed during the current study are not publicly available due to the pilot nature of the study and lack of patient outcome data but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This publication was funded by the National Institute of Mental Health (R34MH109605 and K08MH116119). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author contributions

ARL and EM2 developed the overarching scientific aims and design of the project. ARL,EM2, FFL, and KL developed the clinical training and consultation content with support from SD. AL, JC, HC, ARL, EM2, FFL, and KL iteratively developed the online training. FFL and KL conducted the post-training consultation. JC, HC, and EM1 conducted all data collection and cleaning and supported analyses. KMK developed and executed the data analysis plan, interpreted implementation outcome and mechanism data, and drafted the results. EHC supported writing all paper sections. All authors contributed to the development, drafting, or review of the manuscript. All authors approved the final manuscript.

Acknowledgements

None.

Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010 Mar;17(1):1-30. https://doi.org/1111/j.1468-2850.2009.01187.x.
Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clin Psychol Rev. 2010 Jun;30:448-66. https://doi.org/1016/j.cpr.2010.02.005.
Lyon AR, Stirman SW, Kerns SE, Bruns EJ. Developing the mental health workforce: review and application of training approaches from multiple disciplines. Adm Policy Ment Health. 2011 Jul;38(4):238-53. https://doi.org/1007/s10488-010-0331-y.
Valenstein-Mah H, Greer N, McKenzie L, Hansen L, Strom TQ, Wiltsey Stirman S, et al. Effectiveness of training methods for delivery of evidence-based psychotherapies: a systematic review. Implement Sci. 2020 Dec;15:1-7. https://doi.org/1186/s13012-020-00998-w.
Apramian T, Cristancho S, Watling C, Ott M, Lingard L. "They have to adapt to learn": surgeons' perspectives on the role of procedural variation in surgical education. J Surg Educ. 2016 Mar-Apr;73(2):339-47. https://doi.org/10.1016/j.jsurg.2015.10.016.
Frank HE, Becker‐Haimes EM, Kendall PC. Therapist training in evidence‐based interventions for mental health: a systematic review of training approaches and outcomes. Clin Psychol Sci Pract. 2020 Sep;27(3):e12330. https://doi.org/1111/cpsp.12330.
Weisz JR, Thomassin K, Hersh J, Santucci LC, MacPherson HA, Rodriguez GM, et al. (2020). Clinician training, then what? Randomized clinical trial of child STEPs psychotherapy using lower-cost implementation supports with versus without expert consultation. J Consult Clin Psychol. 2020 Dec;88(12):1065-78. https://doi.org/10.1037/ccp0000536.
Edmunds JM, Beidas RS, Kendall PC. Dissemination and implementation of evidence–based practices: training and consultation as implementation strategies Clin Psychol Sci Pract. 2013 Jun;20(2):152-65. https://doi.org/1111/cpsp.12031.
Lyon AR, Pullmann MD, Walker SC, D’Angelo G. Community-sourced intervention programs: review of submissions in response to a statewide call for “promising practices.” Admin Pol Ment Health. 2017 Jan;44(1):16-28. https://doi.org/10.1007/s10488-015-0650-0.
Institute of Medicine, Board on Health Sciences Policy, Committee on Developing Evidence-Based Standards for Psychosocial Interventions for Mental Disorders. Psychosocial interventions for mental and substance use disorders: a framework for establishing evidence-based standards. England MJ, Butler AS, Gonzalez ML, editors. Washington (DC): National Academies Press; 2015. https://doi.org/10.17226/19013.
Cook CR, Lyon AR, Locke J, Waltz T, Powell BJ. Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prev Sci. 2019 Aug;20(6):914-35. https://doi.org/10.1007/s11121-019-01017-1.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 Feb;10:1-14. https://doi.org/10.1186/s13012-015-0209-1.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implementation Science. 2013 Dec;8(1):1-11. https://doi.org/10.1186/1748-5908-8-139.
Lyon AR, Cook CR, Locke J, Davis C, Powell BJ, Waltz TJ. Importance and feasibility of an adapted set of strategies for implementing evidence-based mental health practices in schools. J Sch Psychol. 2019 Oct;76:66-77.https://doi.org/10.1016/j.jsp.2019.07.014.
Awad M, Connors E. Measurement Based Care as a function of ethical practice: the role of supervisors in supporting MBC values. Manuscript in preparation. 2021.
Chafouleas SM, Clonan SM, Vanauken TL. A national survey of current supervision and evaluation practices of school psychologists. Psychol Sch. 2002 May;39(3):317-25. https://doi.org/10.1002/pits.10021.
Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box. Admin Pol Ment Health. 2013 Nov;40(6):439-50. https://doi.org/10.1007/s10488-013-0502-8.
Olson JR, Lucy M, Kellogg MA, Schmitz K, Berntson T, Stuber J, et al. What happens when training goes virtual? Adapting training and technical assistance for the school mental health workforce in response to COVID-19. School Ment Health. 2021 Mar;13(1):160-73. https://doi.org/10.1007/s12310-020-09401-x.
Weingardt KR, Cucciare MA, Bellotti C, Lai WP. A randomized trial comparing two models of web-based training in cognitive–behavioral therapy for substance abuse counselors. J Subst Abuse Treat. 2009 Oct;37(3):219-27. https://doi.org/10.1016/j.jsat.2009.01.002.
Dimeff LA, Koerner K, Woodcock EA, Beadnell B, Brown MZ, Skutch JM, et al. Which training method works best? A randomized controlled trial comparing three methods of training clinicians in dialectical behavior therapy skills. Behav Res Ther. 2009 Nov;47(11):921-30. https://doi.org/10.1016/j.brat.2009.07.011.
Hubley S, Woodcock EA, Dimeff LA, Dimidjian S. Disseminating behavioural activation for depression via online training: preliminary steps. Behav Cogn Psychother. 2015 Mar;43(2):224-38. https://doi.org/10.1017/S1352465813000842.
Dimeff LA, Harned MS, Woodcock EA, Skutch JM, Koerner K, Linehan MM. Investigating bang for your training buck: a randomized controlled trial comparing three methods of training clinicians in two core strategies of dialectical behavior therapy. Behav Ther. 2015 May;46(3):283-95. http://doi.org/10.1016/j.beth.2015.01.001.
Beidas RS, Koerner K, Weingardt KR, Kendall PC. Training research: practical recommendations for maximum impact. Adm Policy Ment Health. 2011 Jul;38(4):223-37. https://doi.org/10.1007/s10488-011-0338-z.
Beidas RS, Edmunds JM, Marcus SC, Kendall PC. Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. Psychiatr Serv. 2012 Jul;63(7):660-5. https://doi.org/10.1176/appi.ps.201100401.
Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F, et al. Implementation research: a synthesis of the literature. 2005. Retrieved from http://nirn.fmhi.usf.edu.
Olmstead T, Carroll KM, Canning-Ball M, Martino S. Cost and cost-effectiveness of three strategies for training clinicians in motivational interviewing. Drug Alcohol Depend. 2011 Jul;116(1-3):195-202. https://doi.org/10.1016/j.drugalcdep.2010.12.015.
Edmunds JM, Read KL, Ringle VA, Brodman DM, Kendall PC, Beidas RS. Sustaining clinician penetration, attitudes and knowledge in cognitive-behavioral therapy for youth anxiety. Implement Sci. 2014 Jul;9:1-12. https://doi.org/10.1186/s13012-014-0089-9.
Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018 May;6:136. https://doi.org/10.3389/fpubh.2018.00136.
Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020 Dec;15:1-25. https://doi.org/10.1186/s13012-020-00983-3.
Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health. 2016 Sep;43(5):783-98. https://doi.org/10.1007/s10488-015-0693-2.
Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, McHugh SM, Weiner BJ. Enhancing the impact of implementation strategies in healthcare: a research agenda. Front Public Health. 2019 Jan; 22;7:3. https://doi.org/10.3389/fpubh.2019.00003.
Williams NJ, Beidas RS. Annual research review: the state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field. J Child Psychol Psychiatry. 2019 Apr;60(4):430-50. https://doi.org/10.1111/jcpp.12960.
Wong SC, McEvoy MP, Wiles LK, Lewis LK. Magnitude of change in outcomes following entry-level evidence-based practice training: a systematic review. Int J Med Educ. 2013 Jun;4:107-14. https://doi.org/10.5116/ijme.51a0.fd25.
Higa CK, Chorpita BF. Evidence-based therapies: translating research into practice. In: Steele RG, Elkin TD, Roberts MC, editors. Handbook of evidence-based therapies for children and adolescents. Boston, MA: Springer; 2008. p. 45-61.
Siqueland L, Crits-Christoph P, Barber JP, Butler SF, Thase M, Najavits L, et al. The role of therapist characteristics in training effects in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. J Psychother Pract Res. 2000;9(3):123-130.
Beidas RS, Cross W, Dorsey S. Show me, don’t tell me: behavioral rehearsal as a training and analogue fidelity tool. Cogn Behav Pract. 2014 Feb;21(1):1-11. https://doi.org/10.1016/j.cbpra.2013.04.002.
Nelson TD, Steele RG. Predictors of practitioner self-reported use of evidence-based practices: practitioner training, clinical setting, and attitudes toward research. Adm Policy Ment Health. 2007 Jul;34(4):319-30. https://doi.org/10.1007/s10488-006-0111-x.
Nelson MM, Shanley JR, Funderburk BW, Bard E. Therapists’ attitudes toward evidence-based practices and implementation of parent–child interaction therapy. Child Maltreat. 2012 Feb;17(1):47-55. https://doi.org/10.1177/1077559512436674.
Pemberton JR, Conners-Burrow NA, Sigel BA, Sievers CM, Stokes LD, Kramer TL. Factors associated with clinician participation in TF-CBT post-workshop training components. Adm Policy Ment Health. 2017 Jul;44(4):524-33. https://doi.org/10.1007/s10488-015-0677-2.
Joyce BR, Showers B. Student achievement through staff development. Presented at the: Association for Supervision and Curriculum Development; 2002; Alexandria, VA. https://www.unrwa.org/sites/default/files/joyce_and_showers_coaching_as_cpd.pdf. Accessed April 23, 2020.
Mannix KA, Blackburn IM, Garland A, Gracie J, Moorey S, Reid B, et al. Effectiveness of brief training in cognitive behaviour therapy techniques for palliative care practitioners. Palliat Med. 2006 Sep;20(6):579-84. https://doi.org/10.1177/0269216306071058.
Lyon AR, Dorsey S, Pullmann M, Silbaugh-Cowdin J, Berliner L. Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Adm Policy Ment Health. 2015 Jan ;42(1):47-60. https://doi.org/10.1007/s10488-014-0543-7.
Fortney JC, Unützer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, et al. A tipping point for measurement-based care. Focus. 2018 Jul;16(3):341-50. https://doi.org/10.1176/appi.focus.16303.
Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. 2015 Feb;22(1):49-59. https://doi.org/10.1016/j.cbpra.2014.01.010.
Connors EH, Douglas S, Jensen-Doss A, Landes SJ, Lewis CC, McLeod BD, et al. What gets measured gets done: how mental health agencies can leverage measurement-based care for better patient care, clinician supports, and organizational goals. Adm Policy Ment Health. 2021 Mar;48(2):250-65. https://doi.org/10.1007/s10488-020-01063-w.
Lyon AR, Pullmann MD, Dorsey S, Martin P, Grigore AA, Becker EM, et al. Reliability, validity, and factor structure of the Current Assessment Practice Evaluation-Revised (CAPER) in a national sample. J Behav Health Serv Res. 2019 Jan;46(1):43-63. https://doi.org/10.1007/s11414-018-9621-z.
Jensen-Doss A, Smith AM, Becker-Haimes EM, Ringle VM, Walsh LM, Nanda M, et al. Individualized progress measures are more acceptable to clinicians than standardized measures: results of a national survey. Adm Policy Ment Health. 2018 May;45(3):392-403. https://doi.org/10.1007/s10488-017-0833-y.
Lewis CC, Boyd M, Puspitasari A, Navarro E, Howard J, Kassab H, et al. Implementing measurement-based care in behavioral health: A review. JAMA psychiatry. 2019 Mar;76(3):324-35. https://doi.org/10.1001/jamapsychiatry.2018.3329.
Resnick SG, Oehlert ME, Hoff RA, Kearney LK. Measurement-based care and psychological assessment: using measurement to enhance psychological treatment. Psychol Serv. 2020 Aug;17(3):233-37. http://dx.doi.org/10.1037/ser0000491.
Lyon AR, Ludwig K, Wasse JK, Bergstrom A, Hendrix E, McCauley E. Determinants and functions of standardized assessment use among school mental health clinicians: a mixed methods evaluation. Adm Policy Ment Health. 2016 Jan;43(1):122-34. https://doi.org/10.1007/s10488-015-0626-0.
Lyon AR, Ludwig K, Romano E, Leonard S, Vander Stoep A, McCauley E. “If it’s worth my time, I will make the time”: school-based providers’ decision-making about participating in an evidence-based psychotherapy consultation program. Adm Policy Ment Health. 2013 Nov;40(6):467-81. https://doi.org/10.1007/s10488-013-0494-4.
Duong MT, Lyon AR, Ludwig K, Wasse JK, McCauley E. Student perceptions of the acceptability and utility of standardized and idiographic assessment in school mental health. Int J Ment Health Promot. 2016 Jan;18(1):49-63. https://doi.org/10.1080/14623730.2015.1079429.
Duong MT, Bruns EJ, Lee K, Cox S, Coifman J, Mayworm A, et al. Rates of mental health service utilization by children and adolescents in schools and other common service settings: a systematic review and meta-analysis. Adm Policy Ment Health. 2021 May;48(3):420-39. https://doi.org/10.1007/s10488-020-01080-9.
Burns C, Dishman E, Johnson B, Verplank B. “Informance”: Min(d)ing future contexts for scenario-based interaction design. Presented at: BayCHI; 1995 Aug; Palo Alto, CA. Abstract available at http://www.baychi.org/_sg.
Sanetti LM, Collier-Meek MA. Increasing implementation science literacy to address the research-to-practice gap in school psychology. J Sch Psychol. 2019 Oct;76:33-47. https://doi.org/10.1016/j.jsp.2019.07.008.
Connors EH, Arora P, Curtis L, Stephan SH. Evidence-based assessment in school mental health. Cogn Behav Pract. 2015 Feb;22(1):60-73. https://doi.org/10.1016/j.cbpra.2014.03.008.
Kazdin AE. Addressing the treatment gap: a key challenge for extending evidence-based psychosocial interventions. Behav Res Ther. 2017 Jan;88:7-18. https://doi.org/10.1016/j.brat.2016.06.004.
Lancet Global Mental Health Group. Scale up services for mental disorders: a call for action. The Lancet. 2007 Oct;370(9594):1241-52. https://doi.org/10.1016/S0140-6736(07)61242-2.
Connors E, Lawson G, Wheatley-Rowe D, Hoover S. Exploration, preparation, and implementation of standardized assessment in a multi-agency school behavioral health network. Adm Policy Ment Health. 2021 May;48(3):464-81. https://doi.org/10.1007/s10488-020-01082-7.
Lyon AR, Coifman J, Cook H, McRee E, Liu FF, Ludwig K, et al. The Cognitive Walkthough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability. Under review.
Lyon A, Liu FF, Coifman JI, Cook H, King K, Ludwig K, et al. Randomized trial to optimize a brief online training and consultation strategy for measurement-based care in school mental health. Proceedings of the Fifth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2019. Implementation Science. 2020;15: A37. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-020-01033-8#Sec142.
Bruns EJ, Pullmann MD, Nicodimos S, Lyon AR, Ludwig K, Namkung N, et al. Pilot test of an engagement, triage, and brief intervention strategy for school mental health. School Ment Health. 2019 Mar;11(1):148-62. https://doi.org/10.1007/s12310-018-9277-0.
Stumpf RE, Higa-McMillan CK, Chorpita BF. Implementation of evidence-based services for youth: assessing provider knowledge. Behav Modif. 2009 Jan;33(1):48-65. https://doi.org/10.1177/0145445508322625.
Jensen-Doss A, Haimes EM, Smith AM, Lyon AR, Lewis CC, Stanick CF, et al. Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Adm Policy Ment Health. 2018 Jan;45(1):48-61. https://doi.org/10.1007/s10488-016-0763-0.
R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. 2018.
Pinheiro J, Bates D, DebRoy S, Sarkar D, R Core Team. nlme: linear and nonlinear mixed effects models. R package version 3.1-137. Vienna, Austria: R Foundation for Statistical Computing. 2018.
Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta‐ Clin Psychol Sci Pract. 2003 Sep;10(3):288-301. https://doi.org/10.1093/clipsy.bpg025.
Chorpita BF, Weisz JR, Daleiden EL, Schoenwald SK, Palinkas LA, Miranda J, et al. Long-term outcomes for the Child STEPs randomized effectiveness trial: a comparison of modular and standard treatment designs with usual care. J Consult Clin Psychol. 2013 Dec;81(6):999-1009. https://doi.org/10.1037/a0034200.
Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Miranda J, Bearman SK, et al. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. Arch Gen Psychiatry. 2012 Mar;69(3):274-82. https://doi.org/10.1001/archgenpsychiatry.2011.147.
Dorsey S, Berliner L, Lyon AR, Pullmann MD, Murray LK. A statewide common elements initiative for children’s mental health. J Behav Health Serv Res. 2016 Apr;43(2):246-61. https://doi.org/10.1007/s11414-014-9430-y.
Lyon AR, Stanick C, Pullmann MD. Toward high‐fidelity treatment as usual: evidence‐based intervention structures to improve usual care psychotherapy. Clin Psychol Sci Pract. 2018 Dec;25(4):e12265. https://doi.org/10.1111/cpsp.12265.
Warren JS, Nelson PL, Mondragon SA, Baldwin SA, Burlingame GM. Youth psychotherapy change trajectories and outcomes in usual care: community mental health versus managed care settings. J Consult Clin Psychol. 2010 Apr;78(2):144-55. https://doi.org/10.1037/a0018544.
Adrian M, Lyon AR, Nicodimos S, Pullmann MD, McCauley E. Enhanced “train and hope” for scalable, cost-effective professional development in youth suicide prevention. Crisis. 2018 Jul;39(4):235-46. https://doi.org/10.1027/0227-5910/a000489.
Glasgow RE, Fisher L, Strycker LA, Hessler D, Toobert DJ, King DK, et al. Minimal intervention needed for change: definition, use, and value for improving health and health research. Transl Behav Med. 2014 Mar;4(1):26-33. https://doi.org/10.1007/s13142-013-0232-1.
von Thiele Schwarz U, Aarons GA, Hasson H. The value equation: three complementary propositions for reconciling fidelity and adaptation in evidence-based practice implementation. BMC Health Serv Res. 2019 Dec;19(1):1-10. https://doi.org/10.1186/s12913-019-4668-y.
Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. J Am Acad Child Adolesc Psychiatry. 2008 Oct;47(10):1114-19. https://doi.org/10.1097/CHI.0b013e3181825af8.
Lyon AR, Lewis CC. Designing health information technologies for uptake: development and implementation of measurement feedback systems in mental health service delivery. Adm Policy Ment Health. 2016 May;43(3):344-49. https://doi.org/10.1007/s10488-015-0704-3.

Table 1

Clinician participant demographic, professional, and caseload characteristics
Characteristic	Intervention Group		Control Group
Characteristic	N	%	N	%
Gender
Female	32	86.5%	36	94.7%
Male	5	13.5%	2	5.3%
Latino
Yes	4	10.8%	3	7.9%
No	33	89.2%	35	92.1%
Race
Asian	1	2.7%	0	0.0%
Black or African-American	7	18.9%	6	15.8%
Native Hawaiian or Other Pacific Islander	1	2.7%	0	0.0%
White or Caucasian	25	67.6%	30	78.9%
Multi-racial	3	8.1%	2	5.3%
Age
18 to 24 years old	0	0.0%	1	2.6%
25 to 34 years old	13	35.1%	10	26.3%
35 to 44 years old	13	35.1%	16	42.1%
45 to 54 years old	7	18.9%	8	21.1%
55 to 64 years old	4	10.8%	3	7.9%
Primary Professional Role
School Psychologist	6	16.2%	5	13.2%
School Counselor	2	5.4%	2	5.3%
School Social Worker	9	24.3%	14	36.8%
Mental Health Counselor	15	40.5%	12	31.6%
Other	5	13.5%	5	13.2%
Years in Role
< 5 years	14	37.8%	12	31.6%
5-9 years	12	32.4%	10	26.3%
10-14 years	4	10.8%	6	15.8%
15-19 years	4	10.8%	7	18.4%
20 + years	3	8.1%	3	7.9%
Geography
Midwest	10	27.0%	17	44.7%
Northeast	10	27.0%	6	15.8%
Southeast	12	32.4%	13	34.2%
Southwest	0	0.0%	0	0.0%
West	5	13.5%	2	5.3%
Urbanicity
Rural	5	13.5%	9	23.7%
Suburban	12	32.4%	9	23.7%
Urban	12	32.4%	14	36.8%
Mixed	7	18.9%	6	15.8%
Other	1	2.7%	0	0.0%
Caseload size (# students seen/week)
5 or fewer	1	2.7%	3	7.9%
6 - 9	5	13.5%	8	21.1%
10 - 14	8	21.6%	6	15.8%
15 or more	23	62.2%	21	55.3%
Grade Level of Students in Caseload
Elementary School (K-5)	13	35.1%	16	42.1%
Middle School (6-8)	16	43.2%	19	50.0%
High School (9-12)	20	54.1%	23	60.5%
% Students Receiving Special Education Services in Caseload
None	0	0.0%	1	2.6%
1 – 25 %	12	32.4%	9	23.7%
26 – 50 %	11	29.7%	14	36.8%
51 – 75 %	6	16.2%	6	15.8%
76 – 100 %	8	21.6%	8	21.1%
% English Language Learners in Caseload
None	5	13.5%	7	18.4%
1 – 25 %	21	56.8%	23	60.5%
26 – 50 %	9	24.3%	3	7.9%
51 – 75 %	2	5.4%	4	10.5%
76 – 100 %	0	0.0%	1	2.6%

Table 2

Implementation outcomes and mechanisms across time
Measure	Baseline				Week 8				Week 16				Week 24				Week 32
	BOLT		Control		BOLT		Control		BOLT		Control		BOLT		Control		BOLT		Control
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
Outcomes (MBC Practices)
Standardized Assessment	2.135	0.928	2.128	0.923	1.889	0.809	1.838	0.868	2.152	0.958	1.796	0.837	2.185	0.867	1.952	0.915	2.028	0.810	1.685	0.813
Individualized Assessment	2.000	1.006	2.385	1.085	2.236	1.072	1.914	0.959	2.515	1.189	1.931	0.880	2.431	1.015	1.986	0.887	2.514	1.032	1.959	0.931
Treatment Modification	1.571	0.686	1.897	0.829	1.625	0.659	1.729	0.817	1.652	0.667	1.722	0.815	1.861	0.780	1.743	0.701	1.833	0.665	1.662	0.708
Mechanisms
MBC Knowledge	0.750	0.085	0.725	0.115	0.838	0.101	0.761	0.140	0.821	0.105	0.755	0.142	0.827	0.107	0.750	0.146	0.831	0.110	0.755	0.163
MBC Attitudes
Benefit	4.343	0.473	4.305	0.448	4.358	0.493	4.137	0.403	4.448	0.438	4.236	0.393	4.425	0.462	4.080	0.668	4.417	0.438	4.276	0.451
Harm	2.101	0.605	2.231	0.779	2.069	0.750	2.257	0.789	2.061	0.628	2.264	0.763	2.049	0.649	2.229	0.651	2.076	0.746	2.169	0.651
MBC Skill	3.399	0.797	3.494	0.767	3.660	0.705	3.525	0.654	3.924	0.573	3.476	0.765	3.743	0.648	3.339	0.729	3.903	0.700	3.625	0.641

Table 3

Main Effects of BOLT+PTC On Outcomes Over Time
	Standardized Assessment			Individualized Assessment			Treatment Modification
	b	S.E.	p	b	S.E.	p	b	S.E.	p
Intercept	2.05	0.12		2.28	0.13	<0.001	1.79	0.09
Week (Linear)	-0.04	0.01	<0.001	-0.04	0.01	<0.001	-0.01	0.00	0.07
Week (Quadratic)	0.0094	0.0002	<0.001	0.0011	0.0003	<0.001
BOLT+PTC vs. Control	0.06	0.16	0.71	-0.04	0.18	0.83	-0.11	0.12	0.36
Week (Linear) X BOLT+PTC vs. Control	0.02	0.01	<0.01	0.05	0.01	<0.001	0.01	0.00	0.08
Week (Quadratic) X BOLT+PTC vs. Control	-0.0005	0.0002	0.09	-0.0011	0.00036	<0.001

*Note: Bolded coefficients are significant at p < .05. Coefficients are rounded to the nearest non-zero decimal place.

Table 4

Effects of Consultation Dose on BOLT Outcomes Over Time
	Standardized Assessment			Individualized Assessment			Treatment Modification
	b	S.E.	p	b	S.E.	p	b	S.E.	p
Standardized Assessment Intercept	2.08	0.09		2.24	0.10		1.70	0.07
Week (Linear)	-0.02	0.00	<0.001	-0.01	0.01	0.23	0.00	0.00	0.78
Week (Quadratic)	0.0006	0.0002	<0.01	0.0002	0.0002	0.24	--
BOLT vs. Control	0.03	0.12	0.823	-0.04	0.13	0.77	-0.09	0.09	0.33
2 Week PTC vs. 4/8 Week PTC	0.02	0.15	0.92	-0.29	0.17	0.01	-0.05	0.12	0.66
4 vs. 8 Week PTC	0.00	0.14	0.98	0.06	0.15	0.71	0.03	0.11	0.74
Week (Linear) X BOLT vs. Control	0.018	0.007	<0.01	0.04	0.008	<0.001	0.006	0.003	0.06
Week (Linear) X 2 Week PTC vs. 4/8 Week PTC	-0.016	0.008	0.04	0.0048	0.009	0.61	-0.003	0.004	0.38
Week (Linear) X 4 vs. 8 Week PTC	0.003	0.006	0.65	-0.0016	0.007	0.03	-0.007	0.004	0.05
Week (Quadratic) X BOLT vs. Control	-0.0004	0.0002	0.12	-0.0008	0.0003	<0.01	--
Week (Quadratic) X 2 Week PTC vs. 4/8 Week PTC	0.0006	0.0003	0.04	-0.00004	0.0004	0.93	--
Week (Quadratic) X 4 vs. 8 Week PTC	0.0002	0.0003	0.60	0.0006	0.0003	0.06	--

*Note: Bolded coefficients are significant at p < .05. Coefficients are rounded to the nearest non-zero decimal place.

Download PDF

Journal Publication

published 21 Jul, 2022

Read the published version in Implementation Science Communications →

Version 1

posted

You are reading this latest preprint version

How Low Can You Go? Examining the Effects of Brief Online Training and Post-Training Consultation Dose on Implementation Mechanisms and Outcomes for Measurement-Based Care

Status:

Journal Publication

Version 1

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Discussion

Conclusion And Future Directions

Abbreviations

Declarations

References

Tables

Supplementary Files

Status:

Journal Publication

Version 1