This study provides valuable insights on clinician, supervisor, and administrator preferences for implementation planning in large public behavioral health systems and highlights important directions for future research. Results also illustrate the utility of BWS as a methodology for rigorously and efficiently eliciting stakeholder preferences for implementation strategies in large-scale behavioral health and health systems.
By identifying four distinct subpopulations of clinicians, supervisors, and administrators whose preferences reflected distinct foci for implementation strategies, these findings highlight the heterogeneity of stakeholder preferences and point to the need for a new research agenda that unpacks the relationships between preference, implementation effectiveness, and tailoring of implementation strategies. Even as Segment 1 (35% of the sample) strongly preferred all financial incentive strategies above any other strategy, another group, Segment 4 (26% of the sample), showed no interest in financial incentives, preferring instead consultation with EBP experts, and yet another group, Segment 2 (23% of the sample) exhibited strong preferences for technology-based strategies. These groups were all distinct from Segment 3 (15% of the sample) which preferred an improved waiting room (to help relax patients and prepare them to engage in an EBP-focused session) and viewed any type of clinical consultation as unhelpful. These distinct segments suggest that a one-size-fits-all implementation strategy may not be successful, and certainly will not be preferred, by the majority of stakeholders. Different implementation strategies may need to be matched with these distinct subpopulations. There is growing consensus in implementation science that strategies should be selected and tailored based on contextual factors with regard to the EBP, setting, and individual characteristics (48, 49). Our results highlight stakeholder preference as a potentially important dimension for tailoring implementation strategies and point to the need for research to better understand how preferences influence EBP implementation.
Targeting implementation strategies based on stakeholders’ preferences may result in more successful EBP implementation in at least three ways. First, if implementation strategies are differentially effective for different individuals and contexts, stakeholder preferences may signal which strategy will be most effective for a given situation. This is similar to the idea of precision medicine in which the most effective intervention (i.e., implementation strategy) depends on the characteristics of the specific individual in context. Second, targeting strategies to stakeholders’ preferences may have a general accelerator effect that increases the effectiveness of any strategy compared to its baseline effectiveness due to engagement or buy-in. For example, if participants are more engaged or invested in a strategy and willing to exert more effort to ensure its success because they chose it, this may increase the strategy’s effectiveness. Ideally, research could quantify the magnitude of this ‘preference effect’ to determine how much increase in effectiveness could be expected for any strategy simply by allowing stakeholders to choose it. Third, assuming that some strategies are universally more effective than others, it may be beneficial to understand stakeholders’ preferences so that policymakers and other leaders can identify stakeholders who do not prefer effective strategies and use supplemental interventions (e.g., a readiness strategy) with these individuals prior to or concurrent with the launch of the effective strategy. Finally, it may be that preference has no effect on the outcome of implementation strategies. The identification of four distinct preference subpopulations in this study points to the need for research to determine how preferences relate to implementation effectiveness so that resources devoted to implementing EBPs can be optimized.
Across the full sample, one consistent finding was the overall rejection of performance feedback/social comparison strategies, which were rated lower than all other strategies on average for the full sample and were the lowest rated strategies for 3 out of 4 subpopulations. These findings suggest stakeholders overwhelmingly viewed performance feedback/social comparison strategies as unhelpful for supporting EBP implementation. This is consistent with findings from primary care practices, in which primary care clinicians also disliked strategies using social comparison (42). Future qualitative inquiry could provide valuable insights into why stakeholders view feedback/social comparison strategies as unhelpful. Potential mechanisms include discomfort with receiving information that is misaligned with one’s perception of one’s performance or feeling as though there will be negative consequences for poor performance.
The large preference gap between feedback/social comparison and other strategies, such as financial incentives, which emerged as the most preferred strategy on average in the full sample and the first or second choice strategy for 3 out of 4 subpopulations, raises an important question about the relative effectiveness of high cost financial incentives compared to lower cost performance feedback strategies, both of which have some evidence of effectiveness (43, 44). Comparative effectiveness trials that include cost-effectiveness analyses would aid policymakers in selecting among strategies when there is a mismatch between stakeholders’ preferences versus what is known to be effective (45). The generally favorable view of financial incentives in this sample is not surprising against the backdrop of a publicly-funded behavioral health workforce that is poorly compensated and financially stressed, often employed as independent contractors, and are working within organizations that are also struggling financially (46, 47).
Another issue highlighted by these findings is the question of which stakeholder group’s preferences have the strongest implications for successful implementation. Many systems, such as Philadelphia, focus implementation policies primarily at the program level by designating EBP programs and providing financial incentives to agencies (versus clinicians). This raises the question of to what extent policymakers should focus their attention on the preferences of executives and administrators, who make agency-level decisions about EBP (e.g., whether to respond to agency incentives by implementing an EBP program), versus attending to the preferences of clinicians, who ultimately are responsible for implementing EBPs in direct care. Preliminary evidence regarding pay-for-performance in behavioral healthcare suggests organization-focused financial incentives have minimal impact on practice whereas individual clinician-focused incentives can change provider behavior (44).
Our results are subject to limitations and qualifications. Stated preferences were elicited from a controlled experiment on hypothetical implementation options. Real-world implementation behaviors are complicated by numerous factors not accounted for in our controlled experiment; thus, actual implementation behavior could be different from that predicted by our data. However, several features of the study design were implemented following best practice methods and consequently limit the potential for bias. For example, the scenario and implementation strategies were presented as realistically as possible and the number of questions each respondent answered was limited considering the cognitive burden of choice questions. The specific implementation preferences described by this sample of clinicians and administrators in this large public behavioral health system may not generalize to clinicians in more rural areas or in cities or states that have not exhibited similar support for EBP. Further, the particular set of strategies is likely tied to the structure of the US behavioral healthcare system and likely would not generalize to other countries with different healthcare systems. The use of a motivated volunteer sample of stakeholders, while preserving internal validity, may also limit generalizability and affect the relative proportions in the latent class analysis. Finally, clinician preferences are but one factor in many that should guide the selection of implementation strategies to support EBP in a specific setting.