We found that facility contacts used diverse implementation strategies in response to a policy notice requiring them to complete case reviews for patients at high risk of adverse opioid-related events. Randomization to being required to receive additional centralized oversight if a facility failed to meet an a priori target did not significantly impact the implementation strategies that facilities chose. We also identified several respondent and facility characteristics and implementation strategies that were associated with improved case review implementation in this national opioid safety effort.
While the average facility used 23 implementation strategies, we found that only a few key strategies were associated with case review completion rates. Education and adaptation/tailoring emerged as important implementation strategies in the adjusted models. The other strategies associated with increased case review completion rates included adjusting practices based on regular monitoring and adapting practices as needed, while retaining fidelity to critical components of the implementation effort. The ability to adapt/tailor the efforts to local needs is generally considered to be important in other implementation efforts,22,23 and leadership supported and encouraged adaptation in the STORM implementation effort. Understanding which implementation strategies work when and in what context is the “holy grail” of implementation science and can enhance the efficiency, cost-effectiveness, and effectiveness of implementation more broadly.
In addition to the specific survey-defined strategies that were actively used during implementation, we found that pre-implementation academic detailing was one of the strongest predictors of case review completion. Academic detailing is, in and of itself, an evidence-based, multi-component implementation strategy that includes needs assessments, education, and focused training and has contributed to successful implementation efforts across a number of domains.21,24,25 In VA, academic detailing is typically conducted by clinical pharmacists, who deliver face-to-face, 1-on-1 training. Our findings support the notion that academic detailing may be most effective as a pre-implementation, or “preparation” strategy, as we found that academic detailing measured prior to implementation was associated with the outcome. Engaging in academic detailing may also reflect other site-level contextual factors and could reflect engagement, enthusiasm, or leadership support.
While relatively few implementation strategies were important in predicting case review completion rates, two respondent characteristics were significantly associated with this outcome. These included younger age of the respondent and fewer years in one’s current VA role. This finding is consistent with other studies that have also found that younger and more recently trained clinicians are more likely to be early adopters of innovations.26 Alternatively, this finding could be the result of an unmeasured confounder. For example, younger respondents could be present in sites with bigger teams with more capacity to delegate tasks.
We notably found that randomization to policy notices requiring additional oversight did not positively influence facility case review completion rates. The two notices differed in that the “additional oversight” notices included language that a site would be required to receive additional oversight and support if it did not reach the 97% case review completion. One possible explanation for why the inclusion of this requirement did not affect implementation or case review completion is that we measured implementation strategies and case review completion rates too soon after the policy notice was released, and that all sites needed more time to stand up effective processes to complete the case reviews. A second possibility is that the differences in the policy notices were too minor to make a positive impact.
A third possibility is that the requirement of “additional oversight” was in fact detrimental to implementation success. Though this was possible by chance alone, the facilities randomized to the “additional oversight” policy notice were less likely to reach the 97% threshold of case review completion rates than those with the standard policy notice. If points of contact perceived “additional oversight” to be a threat or negative consequence the effect may have been detrimental, in contrast to a potential notice that included a positive incentive or reward. This is a well-established psychological phenomenon wherein positive reinforcement leads to increased intrinsic motivation more so than negative consequences.27 Another complicating factor is that VA has an additional layer of regional management between health care systems and the national office that would provide additional oversight per the notice. Although regional management was made aware that the notice only required oversight and action planning at a randomized subset of sites per the notice, some chose to globally implement their own oversight and action planning requirements across sites in their region, per their standard practices. Variable regional oversight practices may have minimized effects of centralized national oversight.
We found several key similarities and differences when we compared our results from this survey and our previously published results of a similar survey in a national HCV treatment expansion implementation effort. In both implementation efforts, early success was associated with training/education and tailoring to the context.15,16 We also found that tailoring the survey for the STORM implementation effort allowed us to reduce the number of strategies that we assessed. The information that was learned from these assessments of stakeholder-reported implementation strategies in vastly different implementation efforts speaks to the value of this approach to assessing implementation strategies.
There were also key differences between the national opioid case review and HCV treatment implementation efforts. First, few facilities reported engaging patients in efforts to implement case reviews for very high-risk patients, since this effort was focused on a provider activity. This is in contrast to the HCV effort, where patient-facing strategies were universally used and associated with increased treatment.15,16 1 In the HCV effort, the characteristics of the individual respondents were not associated with the outcomes of interest, while the point of contact demographic characteristics appeared to be important to opioid-related case review completion. These key differences may be explained by the differences between the implementation “ask” in these two efforts. The case review effort could be completed by a single team, since the average facility had 18 very high-risk patients. In contrast, in HCV treatment implementation, facilities were asked to treat hundreds of patients, which may have required coordinated implementation efforts across a range of stakeholders. This demonstrates the importance of understanding the context and the “ask” of an implementation effort when determining which strategies to use and how to interpret strategy survey findings.
We acknowledge that there were some notable limitations of this study. First, implementation strategies were reported by a single individual from each facility and may not have reflected the full scope of what was being done at the facility level. However, we have previously found high interrater reliability between multiple respondents from the same facility in a similar study using a similar survey.16 Finally, we conducted multiple statistical tests using a relatively small number of facilities, which allows us to generate hypotheses but not draw definitive conclusions from the findings. Despite these limitations, this was a national, randomized program evaluation with excellent response rates, and our findings add to a growing body of literature assessing a wide variety of implementation strategies across large-scale implementation efforts.