We found that facility contacts used diverse implementation strategies in response to a policy notice requiring them to complete case reviews for patients prescribed opioids who were estimated to be at high risk for adverse events. Randomization to being required to receive additional centralized oversight if a facility failed to meet an a priori target did not significantly impact the implementation strategies that facilities chose. However, we did identify several respondent and facility characteristics and implementation strategies that were associated with improved case review implementation in this national opioid safety effort.
While the average facility used 23 implementation strategies, we found that only a few key strategies were associated with case review completion rates. Education and adaptation/tailoring emerged as important implementation strategies in the adjusted models. Specific strategies associated with increased case review completion rates included adjusting practices based on regular monitoring and adapting practices as needed, while retaining fidelity to critical components of the implementation effort. The ability to adapt/tailor the efforts to local needs is generally considered to be important in other implementation efforts,26,27 and leadership supported and encouraged adaptation in the STORM implementation effort. Understanding which implementation strategies work when and in what context is the “holy grail” of implementation science and can enhance the efficiency, cost-effectiveness, and effectiveness of implementation more broadly.
In addition to the specific survey-defined strategies that were actively used during implementation, we found that pre-implementation academic detailing was one of the strongest predictors of case review completion. Academic detailing is, in and of itself, an evidence-based, multi-component implementation strategy that includes needs assessments, education, and focused training and has contributed to successful implementation efforts across a number of domains.24,28-31 In VA, academic detailing is typically conducted by clinical pharmacists, who deliver face-to-face, 1-on-1 training. Our findings support the notion that academic detailing is effective as a pre-implementation, or “preparation” strategy, as we found that academic detailing measured prior to implementation was associated with the outcome. Engaging in academic detailing may also reflect other site-level contextual factors such as engagement, enthusiasm, or leadership support.
While relatively few implementation strategies were important in predicting case review completion rates, two respondent characteristics were significantly associated with this outcome. These included younger age of the respondent and fewer years in one’s current VA role. This finding is consistent with other studies that have also found that younger and more recently trained clinicians are more likely to be early adopters of innovations.32 Alternatively, this association could be the result of an unmeasured confounder. Future work should assess how team member characteristics relate to their use of specific implementation strategies and their success.
We notably found that randomization to policy notices requiring additional oversight did not positively influence facility case review completion rates. The two notices differed in that the “additional oversight” notices included language that a site would be required to receive additional oversight and support if it did not reach the 97% case review completion. One possible explanation for why the inclusion of this requirement did not affect implementation or case review completion is that we measured implementation strategies and case review completion rates too soon after the policy notice was released, and that all sites needed more time to stand up effective processes to complete the case reviews. A second possibility is that the differences in the policy notices were too minor to make a positive impact. A third possibility is that the requirement of “additional oversight” was in fact detrimental to implementation success. Though this was possible by chance alone, the facilities randomized to the “additional oversight” policy notice were less likely to reach the 97% threshold of case review completion rates than those with the standard policy notice. If points of contact perceived “additional oversight” to be a threat or negative consequence the effect may have been detrimental, in contrast to a potential notice that included a positive incentive or reward. This is a well-established psychological phenomenon wherein positive reinforcement leads to increased intrinsic motivation more so than negative consequences.33 Another complicating factor is that VA has an additional layer of regional management between health care systems and the national office that would provide additional oversight per the notice. Although regional management was made aware that the notice only required oversight and action planning at a randomized subset of sites per the notice, some chose to globally implement their own oversight and action planning requirements across sites in their region, per their standard practices. Variable regional oversight practices may have minimized effects of centralized national oversight.
We found several key similarities and differences when we compared our results from this survey to previously published results from a similar survey conducted in the context of VA’s national HCV elimination program In both implementation efforts, respondents endorsed a similarly high number of strategies, with a median number of 23 vs. 24 implementation strategies in the STORM and HCV efforts, respectively. We also identified strategies associated with more successful implementation that were common to both efforts, including training/education and tailoring to the context.16,17 Tailoring the survey for the STORM implementation effort allowed us to reduce the number of strategies that we assessed (from 73 to 68). That we found useful information from both assessments of stakeholder-reported implementation strategies in vastly different implementation efforts speaks to the value of this approach to assessing implementation strategies An overarching goal in the field of implementation science is to systematize how investigators and healthcare systems choose implementation strategies to address implementation barriers. Moving forward, we hope to develop data around which implementation strategies function best in which settings to address which implementation goals. Then we can test the application of these strategies using randomized experiments or other large, naturalistic, pragmatic operational initiatives like STORM.
There were also key differences between the national opioid case review and HCV treatment implementation efforts. First, few facilities reported engaging patients in efforts to implement case reviews for very high-risk patients, since this effort was focused on a provider activity. This is in contrast to the HCV effort, where patient-facing strategies were universally used and associated with increased treatment.16,1717 In the HCV effort, the characteristics of the individual respondents were not associated with the outcomes of interest, while the point of contact demographic characteristics appeared to be important to opioid-related case review completion. These key differences may be explained by the differences between the implementation “ask” in these two efforts. The case review effort could be completed by a single team, since the average facility had 18 very high-risk patients. In contrast, in HCV treatment implementation, facilities were asked to treat hundreds of patients, which may have required coordinated implementation efforts across a range of stakeholders. This demonstrates the importance of understanding the context and the “ask”, or complexity, of an implementation effort when determining which strategies to use and how to interpret findings. Stemming from Rogers’ work on diffusion of innovations and incorporated into leading implementation frameworks like the Consolidated Framework for Implementation Research, there is good evidence to show that the complexity of the innovation impacts implementation success.34,35 Complexity also likely impacts the choice of implementation strategies. Measuring and evaluating the linkages between complexity, innovation implementation, and implementation strategies could inform how to choose implementation strategies based on the complexity of the innovation so as to improve implementation outcomes.
We acknowledge several notable limitations of this study. First, implementation strategies were reported by a single individual from each facility and may not have reflected the full scope of what was being done at the facility level. However, we have previously found high interrater reliability between multiple respondents from the same facility in a similar study using a similar survey.17 An additional limitation is the potential for contamination across the randomization arms and unblinding to the process of randomization. While facilities were not made aware that two different policy notices were assigned randomly, it is possible that providers could have communicated and become aware of the randomization process, which may have altered the implementation strategies that were chosen. While the survey provides information about whether facilities used a wide range of strategies, it does not address other key elements about the strategies (e.g., intensity, mechanism, fidelity to the strategy), so a key next step is to collect data that better elucidates how the strategies are used. Another limitation is that respondents may not be cognitively distinguish the strategies employed at a particular facility. In addition, they may not appreciate how the implemented strategies were defined or whether the strategies and clusters were useful. We tried to mitigate this limitation by engaging stakeholders in survey development, adding examples relevant to the clinical domain, and clustering the strategies allowed respondents to reach the end of the survey. Nevertheless, future studies could examine whether there were respondent issues and whether our strategies to overcome this limitation are effective. Finally, we conducted multiple statistical tests using a relatively small number of facilities, which allows us to generate hypotheses but not draw definitive conclusions from the findings. Despite these limitations, this was a national, randomized program evaluation with excellent response rates, and our findings add to a growing body of literature assessing a wide variety of implementation strategies across large-scale implementation efforts.