Citizen workshops in public libraries to disseminate and discuss primary care research results: a scaling-up study


 Background

Little is known about engaging patients and stakeholders in the process of scaling up effective knowledge translation interventions targeting the general public. Using an integrated knowledge translation approach, we aimed to scale up and evaluate an effective pilot program of disseminating research results in public libraries.
Methods

We conducted a scaling-up study targeting the general public. Based on our successful pilot project, we co-developed and implemented a larger-scale program of free citizen workshops in public libraries, this time in close research partnership with stakeholders and patient representatives. Citizen workshops, each facilitated by one participating physician and one science communicator, consisted of a 45-min computer-assisted presentation and a 45-min open exchange. Additional scale-up costs included offering financial incentives to stakeholders involved and the purchase of audio-visual equipment. The intervention outcome was knowledge gained. Scale-up outcomes were satisfaction, appropriateness, coverage, time and costs. An evaluation questionnaire was used to collect data of interest. Both quantitative and qualitative analyses were performed.
Results

 The workshop theme chosen by patient and stakeholder representatives was the high prevalence of medication overuse among people over 65 years of age. From April to May 2019, 26 workshops were given in 25 public libraries reaching 362 people. Eighteen participating physicians and six science communicators facilitated the workshops. Participants reported significant knowledge gain (mean difference 2.1, 95% CI 2.0–2.2, P < .001). Median score for overall public satisfaction was 9/10 (IQR 8–10). A high level of appropriateness of the workshops was globally rated by the public participants Coverage was 92.6% of the total number of public libraries targeted. Costs were $6,051.84 CAD for workshop design and $22,935.41 CAD for scaling them up.
Conclusion

This project successfully established a large-scale and successful KT bridge between researchers, clinicians, and citizens via public libraries. This study provides a model for a dissemination practice that benefits the general public by both engaging them in the dissemination process and by targeting them directly.

similar to the target setting. We succeeded in maintaining comparable participation rates for workshops across public libraries, a good indication of the generalizability of our project. The model was easily transferable, matched the values of the target institutions, i.e. libraries, and similar logistics could be applied. Based on our body of evidence, our next step was to investigate how these results would hold on a larger scale, i.e. by targeting more public libraries and delivering more workshops. We expected that, ultimately, reaching a larger public and increasing their knowledge could greatly impact population health.
However, there is no point in scaling up KT intervention projects that are not relevant to their target populations. Knowledge must also be accessible to end-users. To this end, the integrated knowledge translation (IKT) approach has been increasingly adopted in implementation studies (20). IKT aims to gather the views of all stakeholders, including knowledge users, throughout the research process in an inclusive, engaging and interactive manner (21). It is based on research partnership, equitable sharing of power and mutual respect among all stakeholders. The bene ts of this approach have been widely demonstrated in the literature (22). In this scaled up version of our pilot, we planned to engage patient and stakeholders from start to nish, involving men and women at a "high level" of engagement (23), i.e. in choosing the theme, de ning its content, and evaluating its outcomes while maintaining or improving workshop effectiveness.
We therefore aimed to scale up an effective pilot program of disseminating research results to the general public through citizen workshops in public libraries, using an integrated knowledge translation approach, while maintaining delity and with equal or improved effectiveness.

Methods
Since no speci c reporting guidelines for scaling up studies are available, we used an adapted version of the Standards for Reporting Implementation Studies (StaRI) guidelines to report our study (24). We also relied on the TIDieR reporting guidelines for the description of the intervention (25).

Study design
As with the pilot project, we conducted a retrospective pre-post intervention study. Participants selfreported both pre-and post-intervention outcome measures only after the intervention was completed (reducing response shift bias for the outcome measures and less burdensome for participants) (26,27). Using an IKT approach, we adapted the pilot methodology to engage patients and stakeholders throughout this scaling up study.

Context
While the pilot project took place in Quebec City, the scaled-up intervention was extended to Montreal which, like Quebec City, is largely Francophone, so the culture and language were similar. It should be noted, however, that Montreal has a higher immigrant population and contains the city of Westmount, which is more Anglophone. There are also economic differences between localities, Westmount being richer than most districts of Montreal and Quebec City.

Targeted sites and population
Convenience sampling was conducted to select libraries in Quebec City, Montreal and Westmount that were able to include a citizen workshop in their spring 2019 agenda and had the necessary amenities (i.e. video projector, laptop, speaker and room for 30 people). The target population for our study was public library patrons over 18 years old. Their participation was voluntary. In order to maximize the number of participants in the libraries, but also to ensure a variety of pro les (e.g., sex, age, education level) among all participants, libraries were free to schedule the citizen workshops on the dates and daytimes they deemed most convenient (i.e. during the work day or in the evenings; weekdays or weekends).

Planning to scale up the intervention Establishing a committee
This scale-up study began with the formation of the preliminary project steering committee and was informed by the Canadian Institutes of Health Research Integrated and End-of-grant knowledge translation frameworks (28). The proposal for scaling up the pilot intervention, i.e. a presentation with lay and user-friendly content followed by an exchange period, was consensually retained by the committee.

Name
We named this intervention "citizen workshops" because of the strong involvement and responsibility of all stakeholders in the process: primary care researchers would produce results and make them available to physicians, science communicators, and patient and research partners for dissemination to the general public; the public would identify the most relevant results; public libraries would host the workshops and science communicators would facilitate them.

Recruitment
Through a convenience selection, we recruited stakeholder representatives including four experts in patient-oriented research, one science communicator, one primary care physician and two public library o cials. The primary healthcare researcher whose results were selected for dissemination and a patientexpert (i.e. a patient or informal caregiver trained in research) who was a caregiver of a patient facing the health problem addressed, joined the committee for the remaining stages. The library o cials on the committee arranged for invitations to be sent to all public libraries in Montreal, Westmount and Quebec City and then helped to identify the libraries that would host the citizen workshops. In addition to the libraries' usual information channels (programming pamphlets, websites and social media platforms), posters, a dedicated website (29) and radio advertisements, social media platforms (Facebook) and newsletters were used to reach the participants. These means of promotion were designed and approved by all nal committee members including the patient expert.
The committee decided that each citizen workshop would be moderated by a team consisting of a family physician as speaker and a science communicator in charge of facilitating and articulating the message in plain language. Thus, researchers and physicians on the committee issued a letter to be included in primary care professional and research organization newsletters inviting any willing primary care physician (emergency and family medicine) or resident to participate in the project. The only prerequisite was that they had to have good knowledge and practical experience of the health problem addressed. Science communicators were selected by the science communicator member of the steering committee according to their ability to communicate orally in plain language, to lead a constructive discussion with an audience, and to manage the unforeseen (inappropriate questions, speaker forgetting important details or explaining key concepts poorly, technical or operational mishaps) as well as their respectful and empathetic attitude.

IKT strategies
As research partners, the stakeholders contributed to all stages of the process using the following strategies: 1) The members of the preliminary steering committee discussed and approved the aim of the citizen workshops, which was to inform and raise awareness of the research results that would be selected. 2) Then they identi ed the theme for which research results would be disseminated in the citizen workshops: a call for research results was issued to all primary health care research teams in the province with a letter, validated by all committee members, encouraging them to submit their research results for free dissemination. The physicians and researchers on the committee helped identify the best means for disseminating the call for research results throughout the province of Quebec (dissemination networks of primary care research centres, university hospital centres, faculties of medicine, pharmacy and nursing care in Quebec universities). 3) Applications were evaluated exclusively by the committee members who re ected the public or patient voice, i.e., librarians, experts in patient-oriented research and the science communicator. 4) All nal committee members, including the patient-expert and the owner of the selected research results, participated in writing and approving the script for the workshops. 5) Patients and stakeholders on the committee were involved in all stages of implementation, including workshop observation, data collection and outcomes evaluation. 6) Meetings between all actors, including the patient-expert, were organized on a bimonthly basis. A progress report was sent out every two months over 18 months. 7) Patients and stakeholders on the committee were assigned to observe all workshops and to distribute the questionnaires to the participants. 8) Preliminary results of the data analysis for the project were discussed with stakeholders and their comments and suggestions were considered in the nal interpretation of the results.
Implementing the scaled-up intervention Preparation To ensure consistency of citizen workshops across libraries, materials for moderation (i.e. PowerPoint presentation, handouts, notes for each individual moderator) were sent six weeks earlier to all participating moderators. They had two weeks to familiarize themselves with the materials. Then, two one-hour preparatory meetings, spaced two weeks apart, were held by the committee and all participating moderators. During these meetings, the committee gathered moderators' feedback on the documents for consideration and sought their approval.

Workshop content
Each citizen workshop was to be divided into two 45-minute equivalent parts: the rst was a computerassisted presentation of the results and the second was a question-and-answer session. First, an introductory part raised public awareness of the health issue related to the results by de ning terms and providing context. Then followed the actual results of the selected study and a detailed description of their direct impact on the public and potential repercussions on their health. In the question-and-answer session, the science communicator ensured that any questions from the participants remained generic in nature and did not seek a personalised medical opinion. This format was identical to that of the pilot phase workshops except for one major adaptation: the addition of a video clip to the presentation in the rst part of the workshop with the testimony of the patient-expert associated with the project. We made this change because in the pilot study, the workshops generated greater gains in knowledge among young people than older people. We hypothesized that older people might need information presented in a different format to better reach them. On completion of the workshops, participants left with a handout outlining the research results along with additional documents and resources about the health problem addressed. Detailed information about the content of the intervention and the handout can be found in Additional le 1.

Maintaining delity
Except for the addition of the video clip in the presentation, efforts were made to maintain delity to the piloted workshop concept and content. Workshops were given in French in all libraries, even in areas that were predominantly Anglophone. The same content was offered with moderators having comparable pro les. To maintain delity, we had to add some elements to the new contexts: for example, some public libraries did not have projectors for the slide presentations with sound, so we purchased our projection materials.

Scaling up outcomes
Scaling up outcomes were related to selected aspects of acceptability and appropriateness of the citizen workshops among participants, workshop coverage, time and costs.
According to a taxonomy of implementation outcomes by Proctor and al. (30), acceptability is the perception among stakeholders that an innovation is agreeable or satisfactory while appropriateness is the perceived t, relevance, or compatibility of the innovation. These outcomes were measured using 12 closed-ended questions about participants' opinions of the workshop. Acceptability was measured using three questions that focused on the structure of the activity, three on the workshop facilitation and two on whether the workshop met their expectations and whether they would recommend it to others. Participants also indicated their overall satisfaction with the workshop using a discrete eleven-point scale where 0 corresponded to unsatis ed, and 10 to fully satis ed.
Appropriateness, on the other hand, was measured using four questions on the workshop quality and relevance. Answers for all questions except the general satisfaction one, were chosen from a four-point Likert scale (ranging from '1 = totally disagree' to '4 = totally agree'). Qualitative data on participants acceptability was also collected from open-ended questions in the evaluation form.
Coverage was determined by determining the ratio between the numerator, i.e. the number of libraries that hosted the workshops, and the denominator, i.e. the number of libraries targeted for participation.
A partial economic evaluation focusing solely on costs was conducted separately for the workshop design costs and the scaling up costs in order to distinguish between modi able costs related to the scale-up strategy and non-modi able costs related to the intervention. Costs for scaling-up included remuneration of steering committee members, medical moderators, science communicators and patient observers, purchase of the necessary equipment and actual delivery of workshops. Expenses related to designing the citizen workshops included fees for steering committee members for designing and writing the workshop script, as well as lming the video clip incorporated into the presentation.

Intervention outcomes
The main outcome of the intervention was knowledge gain as perceived by participants about the health problem addressed. To assess this, we adapted the self-administered questionnaire used in our pilot study (19). This questionnaire was given to participants at the beginning of the workshop and they were invited to complete it at the end (See Additional le 2). They rated their knowledge using a discrete scale from 0 (very low) to 10 (very high).
Data were also collected on participants' sociodemographic characteristics such as age, sex and highest level of education reached. Finally, other variables pertaining to workshop characteristics were collected by direct observation during their delivery: the time of day during which the workshops were held, the presence or not of the patient expert as observer during the workshop, and whether the speaker was a physician or a resident.

Analysis
First, we performed a descriptive analysis of the participants according to their socio-demographic characteristics and the workshops they attended, their opinions, their levels of satisfaction and knowledge.
Paired T-test was used to compare self-reported pre and post knowledge levels (31,32). Comparative analyses of the knowledge gain were then made according to the characteristics of the participants but also of the workshop in which they participated. To this end, univariate linear regression models of knowledge gain were constructed (33). In order to assess how knowledge gains would vary across the public libraries, comparisons were also made according to the workshops' moderators, (i.e., each facilitator, each speaker and each pair of moderators) using ANOVA test (34). However, given the skewed distributions of knowledge levels and gain, sensitive analyses were performed: rst, the Wilcoxon signedrank comparison test was used to compare before and after median knowledge levels (35). Second, unmatched rank tests on the median and nonparametric multiple comparison were performed using the SAS NPAR1WAY procedure (36).. Statistical signi cance was de ned as p values <0.05 (two-sided test).
All analyses were performed in the SAS software (SAS Institute Inc, Cary, NC, USA, version 9.4).
Qualitative data collected through open-ended questions were transcribed by one author and analyzed using an iterative deductive method discussed with team members. For the economic evaluation, we calculated the sum of expenses separately for the scaling-up strategies and for the design of the citizen workshops. Cost results are presented in Canadian dollars.

Results
Following the committee's call for research results, ve research teams submitted their results. The results selected that responded to public/patient interest, according to the selection committee, addressed the high prevalence of the use of potentially inappropriate medicines (PIMs) among people over 65 years of age in Quebec (37).

Population
A total of 25 libraries, including nine in Quebec City and 16 in Montreal, agreed to host the citizen workshops. From April 4 to May 29, 2019, 26 workshops were offered in Montreal, including one in Westmount and ten in Quebec City, with one library agreeing to host two workshops. Eighteen physicians were mobilized to present the selected ndings and six facilitators were recruited. As a result, 22 distinct pairs of moderators were assembled.
The citizen workshops drew 362 participants with a mean of 13.9 (SD (standard deviation) = 6.0) participants per workshop. The evaluation questionnaire was returned by 320 participants ( Figure A). Table 1 presents the socio-demographic characteristics of participants and characteristics of the workshops in which they participated. The mean age of public participants was 64.8 years (SD=12.5). Women accounted for 71.6% of public participants and half had a university level education (53.8%).
Approximately half (46.9%) attended workshops in the evening, and 18.1% had a patient partner present at their workshop. Most of the participants (87.2%) had a physician as speaker.

Outcomes
Scaling up outcomes

Coverage
Of the 27 public libraries initially planned for the citizen workshops, 25 held workshops, corresponding to a coverage of 92.6%.
Acceptability and appropriateness of citizen workshops, according to public participants The median level of overall satisfaction was 9 (interquartile interval-IQR = 8.0-10) out of 10. Regarding qualitative data, participants pointed out the good quality of the PowerPoint presentation. They particularly liked the inclusion of the interview with the patient partner in the layout of the presentation.
Many participants also perceived and praised the effort of communicating the research results in plain language in the PowerPoint presentation and during workshop facilitation. However, participants expressed some negative impressions, notably that several libraries were open-plan concept and therefore did not have dedicated rooms for this type of activity. Although most participants found the length of the workshops adequate (86%), some found there was not enough time to discuss their concerns. The lowest approval score was obtained for an item that assessed whether their active participation had been encouraged (79.7%). However, for this same item, a high rate of missing responses (13%) was noted. Regarding the moderation of the workshops, most participants reported that the moderators provided an atmosphere conducive to discussion (93.5 %) and gave them useful answers (92.5%). They also appreciated the enthusiasm of the moderators and their complementarity (96.2%).
Finally, most participants felt the workshop met their expectations (91.9%) and 94.1% would recommend the activity to others ( Figure B).
In terms of appropriateness, more than 9 out of 10 participants found that the citizen workshops were accessible to a lay audience and that the information presented to them was clear and of relevant content. A low agreement was obtained, however, on the usefulness of the documentation provided to them (66.9%). This is also the item for which the proportion of missing responses was highest (24.8%). Yet many participants found information in the handouts was too brief and one participant suggested a more substantial document with more information such as examples, useful websites and a detailed outline of the presentation. Overall costs for the project were therefore $28,987.25 CAD.
The duration of the scaling up process using the IKT approach, from the creation of the steering committee to the beginning of the citizen workshops, was 17 months and eight months longer than that of the pilot project.

Intervention outcomes
Knowledge gain As shown in Table 2, in general, participants reported a signi cant gain in knowledge (mean difference (SD) =2.1 (1.7); P < .001). Neither the range of participants' sociodemographic pro les, nor the workshops' characteristics, nor the variety of workshop moderators (as individuals or as pairs) appeared to modify the effect of the workshop on knowledge gain (Table 3). These results were con rmed in our sensitivity analysis (Additional le 3).
No harm was reported from stakeholders or workshop participants.

Discussion
We aimed to evaluate the scaling up of an effective pilot program to disseminate research results through citizen workshops in public libraries. The main departure of the scaled intervention from strict delity to the pilot intervention was that we adopted an IKT approach to ensure that the citizen workshops faithfully re ected the needs and interests of patients and other stakeholders at every step of the intervention. We achieved high coverage of the project to scale up the workshops, which generated high levels of satisfaction among public participants and high levels of acceptability and appropriateness. Participants in the scaled-up citizen workshops also reported an increase in knowledge level of the subject being discussed. These ndings lead us to make the following observations. First, our scaled-up citizen workshops led to an increase in knowledge among participants. Interactive workshops have been established as ideal for sharing knowledge across professional and sectoral boundaries (38). In this project, the interactive aspect was emphasized as much as possible by adding the video clip to the initial format of the workshops to better communicate the patient's perspective.
Although participants in the audience were much older ( mean age 64.8 years) than in the pilot project (mean age 55.0 years), our scaled-up citizen workshops, in addition to being highly satisfying, led to an improvement in knowledge among these participants. These results con rm the importance of designing a more detailed and inclusive format for citizens' workshops, regardless of the topic under discussion, to increase knowledge among all age ranges within the audience. It should also be noted, however, that these results did not allow us to assess the extent to which an increase in knowledge among public participants produced behavioural change. A U.S. study which evaluated the mid-term impact of afterschool nutrition workshops in a public library setting and which targeted adolescents and their parents, a program deemed by the authors to be of low intensity even though it consisted of ve workshops, did not produce any lasting behavioural change after just three months (39). Our citizen workshops, which were one-time events, sought primarily to raise awareness, with behavioural change as an indirect goal. The next step would be to evaluate the immediate and mid-term impacts of the citizen workshops among the public by assessing health outcome data related to the themes both at the time of the workshops and then at intervals afterwards.
Second, adopting an IKT approach improved our scaling up results in the following ways: (a) the involvement of library network stakeholders in identifying participating libraries could explain the high coverage of our scaling-up project; (b) prioritizing the public's perspective to identify the results to be disseminated, adopting a co-constructive approach to designing the workshops, and holding preparatory meetings to allow the workshop moderators to make the content of the message their own are all reasons that could explain our positive results in terms of acceptability and appropriateness among the public; These positive ndings are also consistent with those of our (non-IKT) pilot project. But interestingly, they also turned out to be of equal magnitude (19) despite the differences between the pilot project and the scaling up project. This last observation held also true for the increase in knowledge. This maintenance of improved outcomes despite the change in subject matter, the involvement of various workshop moderators, and the socio-demographic and linguistic differences within the participating public libraries is likely due to the modi cation of the intervention by incorporating an IKT approach from start to nish.
Third, to the best of our knowledge, this is the rst scaling up study to meet such high levels of patient and other stakeholder engagement. Our scaled-up version of the workshop achieved delity in terms of being true to the concept and content from one site to another and largely true to the concept implemented in the pilot trial, with the addition of a patient-designed video clip. However, our pilot project did not use an IKT approach, thus in theory, our scaled-up version of the intervention did not meet the strictest delity requirements of adhering to the intervention as outlined in the original pilot design. This raises an interesting question about knowledge translation. If new knowledge emerges between the pilot program and the scaling up phase (e.g. evidence about the importance of high-level patient engagement), should the scaled-up intervention maintain delity at all costs, or should this new knowledge be integrated into the scaled up version? The science of scaling up must not end up restricting researchers to reproducing interventions at scale that exclude important new knowledge. Indeed, we propose that from now on, the IKT approach should be, as far as possible, an essential and integral dimension of scaling up. At rst glance, IKT appears to be a cumbersome approach since it requires constant consultation and adaptation that could slow down the process of scaling up (40,41). Yet, it ensures that the interventions' effectiveness would not be diluted with scaling up and that the interventions are worth being scaled up because they respond to the real needs and interests of patients and other stakeholders.
In this sense, IKT could also be perceived as a necessary regulator of the upscaling process.
Fourth, as Milat et al. suggest, before scaling up an intervention, evidence of effectiveness should ideally be provided through RTCs (42). In our case, it was impossible to manipulate exposure to the intervention, and so our evidence was from a natural experiment performed in the real world. Therefore, we skipped the RCT step and went straight from our pilot project, a feasibility study, to the scale-up phase. However, the results of the pilot phase had already provided us with information on scalability elements. Scaling up has been taking place, under different names, for several decades (especially in LMICs for quickly stemming the spread of infectious diseases) (43), and current scale-up efforts in LMICs show that scaleup strategies must be su ciently exible to respond to emerging questions (44). Scaling up is still a new science, and as Milat et al. concede, must build exibility in its application to real-world interventions (42).
Fifth, Quebec City, where our pilot took place, is almost unilingually Francophone. Our workshops were scaled up to include libraries in Montreal, which has more immigrants and is more culturally diverse, and Westmount, which is more Anglophone. Although we did not measure these contextual differences in our socio-demographic questionnaires, the positive and consistent effect of the citizen workshops on knowledge gain is a good indication that extending our model to more diverse populations will maintain acceptability and knowledge acquisition levels. However, this does not preclude the importance of adapting to different socio-demographic pro les with scaling up. Further adaptations may depend on the theme addressed, the target population, and the social situation. For instance, the modalities of mass gatherings have changed dramatically with the COVID-19 pandemic. As a result, modi cations in the delivery will have to be made to our citizen workshops to follow public health recommendations.
Finally, we lacked the opportunity to conduct a full economic (cost-effectiveness) analysis. However, our partial cost evaluation could be useful in the future for scaling-up studies, which so far have rarely included economic evaluations (43). In addition, costs are considered an essential reporting item in a proposed guideline for reporting on scaling up studies(45). Full economic evaluations in the real context of scaling up will also help choose e cient strategies involving high-level engagement of patients and stakeholders across the scaling-up process and predicting the economic and human resource costs of further scale-up.
The limitations of our study were: rst, the fact that it had no comparison group. However, our earlier pilot project results helped us understand some of the ndings better. It would be interesting to compare the costs of using an IKT approach to scale up our model to scaling it up without integrating patients and stakeholders, although ethically questionable. Second, participants in the citizen workshops were selfselected citizens who responded to an ad for the workshop. However, self-selection sampling has some advantages: it reduces recruitment time, and self-selected participants are more likely to be committed to take part in the study (e.g. more willing to spend the time lling in the questionnaire) and to provide insights into the theme(46). Nevertheless, we failed to meet the more vulnerable populations with lower literacy levels: half of the public in the workshops were university graduates and therefore not representative of Quebec's overall elderly population literacy level. Third, the data were collected using self-reporting tools; however, the impact of this on the effectiveness analysis should be, if anything, an underestimation of the knowledge gain among participants.

Discussion
We aimed to evaluate the scaling up of an effective pilot program to disseminate research results through citizen workshops in public libraries. The main departure of the scaled intervention from strict delity to the pilot intervention was that we adopted an IKT approach to ensure that the citizen workshops faithfully re ected the needs and interests of patients and other stakeholders at every step of the intervention. We achieved high coverage of the project to scale up the workshops, which generated high levels of satisfaction among public participants and high levels of acceptability and appropriateness. Participants in the scaled-up citizen workshops also reported an increase in knowledge level of the subject being discussed. These ndings lead us to make the following observations. First, our scaled-up citizen workshops led to an increase in knowledge among participants. Interactive workshops have been established as ideal for sharing knowledge across professional and sectoral boundaries (38). In this project, the interactive aspect was emphasized as much as possible by adding the video clip to the initial format of the workshops to better communicate the patient's perspective.
Although participants in the audience were much older ( mean age 64.8 years) than in the pilot project (mean age 55.0 years), our scaled-up citizen workshops, in addition to being highly satisfying, led to an improvement in knowledge among these participants. These results con rm the importance of designing a more detailed and inclusive format for citizens' workshops, regardless of the topic under discussion, to increase knowledge among all age ranges within the audience. It should also be noted, however, that these results did not allow us to assess the extent to which an increase in knowledge among public participants produced behavioural change. A U.S. study which evaluated the mid-term impact of afterschool nutrition workshops in a public library setting and which targeted adolescents and their parents, a program deemed by the authors to be of low intensity even though it consisted of ve workshops, did not produce any lasting behavioural change after just three months (39). Our citizen workshops, which were one-time events, sought primarily to raise awareness, with behavioural change as an indirect goal. The next step would be to evaluate the immediate and mid-term impacts of the citizen workshops among the public by assessing health outcome data related to the themes both at the time of the workshops and then at intervals afterwards.
Second, adopting an IKT approach improved our scaling up results in the following ways: (a) the involvement of library network stakeholders in identifying participating libraries could explain the high coverage of our scaling-up project; (b) prioritizing the public's perspective to identify the results to be disseminated, adopting a co-constructive approach to designing the workshops, and holding preparatory meetings to allow the workshop moderators to make the content of the message their own are all reasons that could explain our positive results in terms of acceptability and appropriateness among the public; These positive ndings are also consistent with those of our (non-IKT) pilot project. But interestingly, they also turned out to be of equal magnitude (19) despite the differences between the pilot project and the scaling up project. This last observation held also true for the increase in knowledge. This maintenance of improved outcomes despite the change in subject matter, the involvement of various workshop moderators, and the socio-demographic and linguistic differences within the participating public libraries is likely due to the modi cation of the intervention by incorporating an IKT approach from start to nish.
Third, to the best of our knowledge, this is the rst scaling up study to meet such high levels of patient and other stakeholder engagement. Our scaled-up version of the workshop achieved delity in terms of being true to the concept and content from one site to another and largely true to the concept implemented in the pilot trial, with the addition of a patient-designed video clip. However, our pilot project did not use an IKT approach, thus in theory, our scaled-up version of the intervention did not meet the strictest delity requirements of adhering to the intervention as outlined in the original pilot design. This raises an interesting question about knowledge translation. If new knowledge emerges between the pilot program and the scaling up phase (e.g. evidence about the importance of high-level patient engagement), should the scaled-up intervention maintain delity at all costs, or should this new knowledge be integrated into the scaled up version? The science of scaling up must not end up restricting researchers to reproducing interventions at scale that exclude important new knowledge. Indeed, we propose that from now on, the IKT approach should be, as far as possible, an essential and integral dimension of scaling up. At rst glance, IKT appears to be a cumbersome approach since it requires constant consultation and adaptation that could slow down the process of scaling up (40,41). Yet, it ensures that the interventions' effectiveness would not be diluted with scaling up and that the interventions are worth being scaled up because they respond to the real needs and interests of patients and other stakeholders. In this sense, IKT could also be perceived as a necessary regulator of the upscaling process.
Fourth, as Milat et al. suggest, before scaling up an intervention, evidence of effectiveness should ideally be provided through RTCs (42). In our case, it was impossible to manipulate exposure to the intervention, and so our evidence was from a natural experiment performed in the real world. Therefore, we skipped the RCT step and went straight from our pilot project, a feasibility study, to the scale-up phase. However, the results of the pilot phase had already provided us with information on scalability elements. Scaling up has been taking place, under different names, for several decades (especially in LMICs for quickly stemming the spread of infectious diseases) (43), and current scale-up efforts in LMICs show that scaleup strategies must be su ciently exible to respond to emerging questions (44). Scaling up is still a new science, and as Milat et al. concede, must build exibility in its application to real-world interventions (42).
Fifth, Quebec City, where our pilot took place, is almost unilingually Francophone. Our workshops were scaled up to include libraries in Montreal, which has more immigrants and is more culturally diverse, and Westmount, which is more Anglophone. Although we did not measure these contextual differences in our socio-demographic questionnaires, the positive and consistent effect of the citizen workshops on knowledge gain is a good indication that extending our model to more diverse populations will maintain acceptability and knowledge acquisition levels. However, this does not preclude the importance of adapting to different socio-demographic pro les with scaling up. Further adaptations may depend on the theme addressed, the target population, and the social situation. For instance, the modalities of mass gatherings have changed dramatically with the COVID-19 pandemic. As a result, modi cations in the delivery will have to be made to our citizen workshops to follow public health recommendations.
Finally, we lacked the opportunity to conduct a full economic (cost-effectiveness) analysis. However, our partial cost evaluation could be useful in the future for scaling-up studies, which so far have rarely included economic evaluations (43). In addition, costs are considered an essential reporting item in a proposed guideline for reporting on scaling up studies(45). Full economic evaluations in the real context of scaling up will also help choose e cient strategies involving high-level engagement of patients and stakeholders across the scaling-up process and predicting the economic and human resource costs of further scale-up.
The limitations of our study were: rst, the fact that it had no comparison group. However, our earlier pilot project results helped us understand some of the ndings better. It would be interesting to compare the costs of using an IKT approach to scale up our model to scaling it up without integrating patients and stakeholders, although ethically questionable. Second, participants in the citizen workshops were selfselected citizens who responded to an ad for the workshop. However, self-selection sampling has some advantages: it reduces recruitment time, and self-selected participants are more likely to be committed to take part in the study (e.g. more willing to spend the time lling in the questionnaire) and to provide insights into the theme(46). Nevertheless, we failed to meet the more vulnerable populations with lower literacy levels: half of the public in the workshops were university graduates and therefore not representative of Quebec's overall elderly population literacy level. Third, the data were collected using self-reporting tools; however, the impact of this on the effectiveness analysis should be, if anything, an underestimation of the knowledge gain among participants.

Conclusions
This project successfully established a large-scale and successful KT bridge between researchers, clinicians, and citizens via public libraries. We found that scaling up a program of citizen workshops in public libraries showed high levels of knowledge gain, content appropriateness, and acceptability. The addition of an IKT approach involving patients and other stakeholders as research partners throughout the process and remunerating them improved the nal product without harming scale-up outcomes. These ndings highlight that an IKT approach and patient-oriented research should no longer be optional.
This study provides a model for a dissemination practice that bene ts the general public by targeting and directly engaging them in the dissemination process. Public libraries are free and power-neutral educational institutions, and this simple and reproducible intervention is a ground-breaking knowledge translation model. Missing data --- PIMs= potentially inappropriate medicines; before = before citizen workshop; after = after citizen workshop; DM = mean difference; SD = standard deviation; * N=276 (after deletion of observations with missing variables) ** Gain is the difference between knowledge level after and knowledge level before the citizen workshop *** Paired T-test was used to compare mean knowledge levels of PIMs, before and after.