SOP was designed as a policy intervention for the Canadian opioid crisis, using best practices in CHPE as a means for driving meaningful change at multiple levels of outcomes. We aimed to offer a program that a) targeted family physicians, b) was accessible regardless of geography, c) was free specifically of pharmaceutical industry bias, d) was interactive, and e) was relevant to practice. This evaluation strongly suggests that through 2014–2017, SOP was delivered as intended along multiple implementation outcomes including reach, dose, fidelity and participant responsiveness. Though sustainability was not formally assessed by this evaluation, sustainability is suggested by the fact that the program was able to continuously run throughout the study period without any external funding.
Family physicians, who are responsible for the majority of long-term and high-dose opioid prescribing, were disproportionately represented in the program. Likewise, participation in the program was representative of the geographic spread of physicians in Ontario, despite the program being delivered virtually from a major urban academic medical centre. This was accomplished in the context of Ontario’s vast geography and known issues of poor access to high-speed internet in rural and remote communities (46). Overall, physician participants were not reflective of the gender mix of Ontario physicians. However, this was mostly driven by participants with medical regulatory involvement, who very much skewed male and also as having more years in practice. This cohort of participants with medical regulatory involvement is reflective of patterns in Ontario that have been identified with respect to potentially problematic opioid prescribing (27) and also patterns of medical regulatory referral to opioid education programs (33).
The program was rated as highly relevant to practice, which partly explains the very high engagement and completion rates. The slightly lower completion rates amongst specialist physicians and family physicians with focused practices, such as in emergency medicine, may in part be driven by lower relevance to practice. It should be noted, however, that completion rates amongst these groups was still very high compared to internet-delivered CHPE norms (47–49). Another reason for lower completion rates might also be less predictable clinical schedules for certain specialists. Importantly, medical regulator involvement was not an important driver of webinar completion.
Out of province completion rates were also high at 75.0%, though not as high as Ontario participants. This may be driven by lower relevance to practice as some of the program content is focused on epidemiological and practical issues specific to Ontario. However, a more practical reason for this difference may be because of time zone differences making it challenging for working professionals to participate in the webinars. Notably, there was a significantly lower completion rate for participants in Western time zones at two or more hours from Ontario, which may have caused the webinar time to conflict with normal clinical hours for participants. The completion rate for those less than a two-hour time difference was comparable to Ontario participants.
The scalability of the program is suggested by the variable size of the webinars, with the largest including 74 simultaneous learners. We identified no systematic variation in time for active learning or relevance to practice based on the size of the webinars. This suggests that fidelity to the program is maintained even with very large numbers of participants. Overall, over a short period, the program was able to reach more than 1% of the 32,055 strong Ontario physician workforce. The demonstrated geographic reach of the program and the potential for scalability suggest that this program is a good model for reaching a critical mass of prescribers to drive population level changes in opioid utilization.
While encouragement or requirement for participation with the medical regulator was not an important driver of webinar completion, it did contribute to the likelihood of workshop participation. Webinar completion was also an important driving factor for workshop participation. Geography played a role in workshop participation as well since non-Ontario participants were less likely to participate in the workshop, which was held in Toronto and thus at least several hours away. The cost of the workshop may have also been prohibitive for some participants given the cumulative costs of registration, travel and lost clinical income. We also noted that, while the difference was not statistically significant, rural Ontario physicians were half as likely as their urban counterparts to participate in the workshop.
As CHPE evaluation has moved increasingly towards outcome-based approaches (15) with a preference for “higher-level” outcomes such as patient-level and population-level outcomes, there has been a related discounting of implementation outcomes such as participation and satisfaction (50). Indeed Moore et al.’s updated framework refers to three different kinds of assessment, namely summative, performance and impact. Importantly, however, this framework ignores assessment of implementation. This may be because educational interventions are not commonly conceptualized as complex interventions that are delivered in complex and dynamic health system and policy contexts – all of which can affect program delivery and structure (implementation) and thus program effects. Thus, rigorous implementation evaluations can be used to determine how the program was actually carried out and whether it was carried out as intended. While the implementation of programs as intended is no guarantee of effectiveness, these data are key to then informing subsequent effectiveness and impact evaluations and also to assessing program theory. Having conducted this implementation evaluation, further evaluation of SOP is now called for to assess effectiveness and impact. To our knowledge, this study is one of few examples of an opioid prescribing CHPE evaluation that has formally assessed implementation outcomes using an evaluation framework for complex interventions together with a CHPE outcome model. Barth et al. (51) describe the use of the Medical Research Council complex intervention framework to develop and evaluate, in a step-wise manner, an academic detailing intervention to improve use of a prescription drug monitoring program (PDMP). A subsequent study evaluates physician self-reports of PDMP utilization – namely a performance (education) or effectiveness (complex intervention) outcome (52). Other opioid prescribing CHPE programs have assessed implementation measures of participation and satisfaction (53–57) but have not directly related these measures to program theory, nor have they used these implementation measures to then inform effectiveness or impact outcomes.
Overall this implementation evaluation adds further support to the feasibility of delivering multicomponent CHPE programs virtually to increase reach, scalability and thus potentially effectiveness and impact (58–60).
There are several important limitations to this study. First, this study was conducted retrospectively using data that were collected for both evaluative purposes but also for administrative purposes, such as for tracking participation for accreditation reporting. We did have to exclude 17 participants (3.2%) of the sample due to incomplete participation data. This was a small enough number that it was unlikely to significantly bias results. Likewise, demographic data was for the most part complete. We could, for example, not determine rurality for only two of the 400 Ontario physician participants. Also, it is important to note that the evaluative data collected (e.g. relevance to practice and amount of interactivity) was defined prior to the delivery of the program and did reflect attempts to assess underlying program logic. The second data limitation relates to the anonymous nature of the evaluative data. These were kept anonymous as per norms in CHPE to allow participants to freely share their evaluative assessments. However, this did not allow us to link evaluative statements to particular participants and then analyze these by demographic factors. This could be rectified in future evaluations by using, for example, a linking identifier that blinds the scientific planning committee to the identity of participants but allows evaluators to link evaluations to the demographic characteristics of specific de-identified program participants. Likewise, the webinar evaluation data response rates were moderate at 51% which would introduce an unknown bias to these data. Since these responses were anonymous, it is not possible to further assess the nature of this possible bias. However, the consistency of these evaluative responses between the webinars and the consistency of the responses with the workshop data which had an excellent response rate provides confidence that these data are reflective of the entire participant population. Third, the available data did not allow for a direct inquiry into the posited drivers of change captured in the logic model, such as that the SOP structure facilitates the creation of a virtual community of learning and practice. Qualitative inquiry using interviews or focus groups of program participants and facilitators would be well suited to better assess this aspect of the program.