Following the committee's call for research results, five research teams submitted their results. The results selected that responded to public/patient interest, according to the selection committee, addressed the high prevalence of the use of potentially inappropriate medicines (PIMs) among people over 65 years of age in Quebec(37).
Population
A total of 25 libraries, including nine in Quebec City and 16 in Montreal, agreed to host the citizen workshops. From April 4 to May 29, 2019, 26 workshops were offered in Montreal, including one in Westmount and ten in Quebec City, with one library agreeing to host two workshops. Eighteen physicians were mobilized to present the selected findings and six facilitators were recruited. As a result, 22 distinct pairs of moderators were assembled.
The citizen workshops drew 362 participants with a mean of 13.9 (SD (standard deviation) = 6.0) participants per workshop. The evaluation questionnaire was returned by 320 participants (Figure A). Table 1 presents the socio-demographic characteristics of participants and characteristics of the workshops in which they participated. The mean age of public participants was 64.8 years (SD=12.5). Women accounted for 71.6% of public participants and half had a university level education (53.8%). Approximately half (46.9%) attended workshops in the evening, and 18.1% had a patient partner present at their workshop. Most of the participants (87.2%) had a physician as speaker.
Outcomes
Scaling up outcomes
Coverage
Of the 27 public libraries initially planned for the citizen workshops, 25 held workshops, corresponding to a coverage of 92.6%.
Acceptability and appropriateness of citizen workshops, according to public participants
The median level of overall satisfaction was 9 (interquartile interval–IQR = 8.0–10) out of 10. Regarding qualitative data, participants pointed out the good quality of the PowerPoint presentation. They particularly liked the inclusion of the interview with the patient partner in the layout of the presentation. Many participants also perceived and praised the effort of communicating the research results in plain language in the PowerPoint presentation and during workshop facilitation. However, participants expressed some negative impressions, notably that several libraries were open-plan concept and therefore did not have dedicated rooms for this type of activity. Although most participants found the length of the workshops adequate (86%), some found there was not enough time to discuss their concerns. The lowest approval score was obtained for an item that assessed whether their active participation had been encouraged (79.7%). However, for this same item, a high rate of missing responses (13%) was noted. Regarding the moderation of the workshops, most participants reported that the moderators provided an atmosphere conducive to discussion (93.5 %) and gave them useful answers (92.5%). They also appreciated the enthusiasm of the moderators and their complementarity (96.2%). Finally, most participants felt the workshop met their expectations (91.9%) and 94.1% would recommend the activity to others (Figure B).
In terms of appropriateness, more than 9 out of 10 participants found that the citizen workshops were accessible to a lay audience and that the information presented to them was clear and of relevant content. A low agreement was obtained, however, on the usefulness of the documentation provided to them (66.9%). This is also the item for which the proportion of missing responses was highest (24.8%). Yet many participants found information in the handouts was too brief and one participant suggested a more substantial document with more information such as examples, useful websites and a detailed outline of the presentation.
Cost and time
Workshop design
Sixteen people were mobilized to participate in the committee. Regarding the design of citizen workshops, costs were mainly the fees of the science communicator member of the steering committee for the writing of the workshop’s script, as well as those of the patient expert for the shooting of her video clip embedded in the presentation. These costs were $6,051.84 CAD. The script revision and the video clip editing were free, as they were performed by other members of the steering committee with the tools already at their disposal in their workplaces.
Scaled up workshop delivery
Again, none of the steering committee members billed for their time since they were professionals who were already paid in their respective workplaces except the science communicator and the patient expert. Their fees regarding the scale-up were $3511.05 CAD. A software was purchased for the posters and the website creation at a cost of $453.10 CAD. The preparatory meetings for the scaled-up workshops, in terms of travel, per diem, and food, cost $4380.12 CAD. For the scaled-up delivery of the citizen workshops, seven external observers were mobilized in addition to the 24 moderators (18 doctors and 6 facilitators). Their per diem, travel and accommodation expenses totalled $13,620.65 CAD. Material used during the workshops (office supplies, recorders, pointers) was evaluated at $970.49 CAD. Total costs for scaling up the intervention were therefore $22,935.41 CAD.
Overall costs for the project were therefore $28,987.25 CAD.
The duration of the scaling up process using the IKT approach, from the creation of the steering committee to the beginning of the citizen workshops, was 17 months and eight months longer than that of the pilot project.
Intervention outcomes
Knowledge gain
As shown in Table 2, in general, participants reported a significant gain in knowledge (mean difference (SD) =2.1 (1.7); P < .001). Neither the range of participants' sociodemographic profiles, nor the workshops' characteristics, nor the variety of workshop moderators (as individuals or as pairs) appeared to modify the effect of the workshop on knowledge gain (Table 3). These results were confirmed in our sensitivity analysis (Additional file 3).
No harm was reported from stakeholders or workshop participants.
Discussion
We aimed to evaluate the scaling up of an effective pilot program to disseminate research results through citizen workshops in public libraries. The main departure of the scaled intervention from strict fidelity to the pilot intervention was that we adopted an IKT approach to ensure that the citizen workshops faithfully reflected the needs and interests of patients and other stakeholders at every step of the intervention. We achieved high coverage of the project to scale up the workshops, which generated high levels of satisfaction among public participants and high levels of acceptability and appropriateness. Participants in the scaled-up citizen workshops also reported an increase in knowledge level of the subject being discussed. These findings lead us to make the following observations.
First, our scaled-up citizen workshops led to an increase in knowledge among participants. Interactive workshops have been established as ideal for sharing knowledge across professional and sectoral boundaries (38). In this project, the interactive aspect was emphasized as much as possible by adding the video clip to the initial format of the workshops to better communicate the patient's perspective. Although participants in the audience were much older ( mean age 64.8 years) than in the pilot project (mean age 55.0 years), our scaled-up citizen workshops, in addition to being highly satisfying, led to an improvement in knowledge among these participants. These results confirm the importance of designing a more detailed and inclusive format for citizens' workshops, regardless of the topic under discussion, to increase knowledge among all age ranges within the audience. It should also be noted, however, that these results did not allow us to assess the extent to which an increase in knowledge among public participants produced behavioural change. A U.S. study which evaluated the mid-term impact of after-school nutrition workshops in a public library setting and which targeted adolescents and their parents, a program deemed by the authors to be of low intensity even though it consisted of five workshops, did not produce any lasting behavioural change after just three months (39). Our citizen workshops, which were one-time events, sought primarily to raise awareness, with behavioural change as an indirect goal. The next step would be to evaluate the immediate and mid-term impacts of the citizen workshops among the public by assessing health outcome data related to the themes both at the time of the workshops and then at intervals afterwards.
Second, adopting an IKT approach improved our scaling up results in the following ways: (a) the involvement of library network stakeholders in identifying participating libraries could explain the high coverage of our scaling-up project; (b) prioritizing the public's perspective to identify the results to be disseminated, adopting a co-constructive approach to designing the workshops, and holding preparatory meetings to allow the workshop moderators to make the content of the message their own are all reasons that could explain our positive results in terms of acceptability and appropriateness among the public; These positive findings are also consistent with those of our (non-IKT) pilot project. But interestingly, they also turned out to be of equal magnitude (19) despite the differences between the pilot project and the scaling up project. This last observation held also true for the increase in knowledge. This maintenance of improved outcomes despite the change in subject matter, the involvement of various workshop moderators, and the socio-demographic and linguistic differences within the participating public libraries is likely due to the modification of the intervention by incorporating an IKT approach from start to finish.
Third, to the best of our knowledge, this is the first scaling up study to meet such high levels of patient and other stakeholder engagement. Our scaled-up version of the workshop achieved fidelity in terms of being true to the concept and content from one site to another and largely true to the concept implemented in the pilot trial, with the addition of a patient-designed video clip. However, our pilot project did not use an IKT approach, thus in theory, our scaled-up version of the intervention did not meet the strictest fidelity requirements of adhering to the intervention as outlined in the original pilot design. This raises an interesting question about knowledge translation. If new knowledge emerges between the pilot program and the scaling up phase (e.g. evidence about the importance of high-level patient engagement), should the scaled-up intervention maintain fidelity at all costs, or should this new knowledge be integrated into the scaled up version? The science of scaling up must not end up restricting researchers to reproducing interventions at scale that exclude important new knowledge. Indeed, we propose that from now on, the IKT approach should be, as far as possible, an essential and integral dimension of scaling up. At first glance, IKT appears to be a cumbersome approach since it requires constant consultation and adaptation that could slow down the process of scaling up (40, 41). Yet, it ensures that the interventions’ effectiveness would not be diluted with scaling up and that the interventions are worth being scaled up because they respond to the real needs and interests of patients and other stakeholders. In this sense, IKT could also be perceived as a necessary regulator of the upscaling process.
Fourth, as Milat et al. suggest, before scaling up an intervention, evidence of effectiveness should ideally be provided through RTCs (42). In our case, it was impossible to manipulate exposure to the intervention, and so our evidence was from a natural experiment performed in the real world. Therefore, we skipped the RCT step and went straight from our pilot project, a feasibility study, to the scale-up phase. However, the results of the pilot phase had already provided us with information on scalability elements. Scaling up has been taking place, under different names, for several decades (especially in LMICs for quickly stemming the spread of infectious diseases)(43), and current scale-up efforts in LMICs show that scale-up strategies must be sufficiently flexible to respond to emerging questions (44). Scaling up is still a new science, and as Milat et al. concede, must build flexibility in its application to real-world interventions (42).
Fifth, Quebec City, where our pilot took place, is almost unilingually Francophone. Our workshops were scaled up to include libraries in Montreal, which has more immigrants and is more culturally diverse, and Westmount, which is more Anglophone. Although we did not measure these contextual differences in our socio-demographic questionnaires, the positive and consistent effect of the citizen workshops on knowledge gain is a good indication that extending our model to more diverse populations will maintain acceptability and knowledge acquisition levels. However, this does not preclude the importance of adapting to different socio-demographic profiles with scaling up. Further adaptations may depend on the theme addressed, the target population, and the social situation. For instance, the modalities of mass gatherings have changed dramatically with the COVID-19 pandemic. As a result, modifications in the delivery will have to be made to our citizen workshops to follow public health recommendations.
Finally, we lacked the opportunity to conduct a full economic (cost-effectiveness) analysis. However, our partial cost evaluation could be useful in the future for scaling-up studies, which so far have rarely included economic evaluations(43). In addition, costs are considered an essential reporting item in a proposed guideline for reporting on scaling up studies(45). Full economic evaluations in the real context of scaling up will also help choose efficient strategies involving high-level engagement of patients and stakeholders across the scaling-up process and predicting the economic and human resource costs of further scale-up.
The limitations of our study were: first, the fact that it had no comparison group. However, our earlier pilot project results helped us understand some of the findings better. It would be interesting to compare the costs of using an IKT approach to scale up our model to scaling it up without integrating patients and stakeholders, although ethically questionable. Second, participants in the citizen workshops were self-selected citizens who responded to an ad for the workshop. However, self-selection sampling has some advantages: it reduces recruitment time, and self-selected participants are more likely to be committed to take part in the study (e.g. more willing to spend the time filling in the questionnaire) and to provide insights into the theme(46). Nevertheless, we failed to meet the more vulnerable populations with lower literacy levels: half of the public in the workshops were university graduates and therefore not representative of Quebec's overall elderly population literacy level. Third, the data were collected using self-reporting tools; however, the impact of this on the effectiveness analysis should be, if anything, an underestimation of the knowledge gain among participants.