PROLIFERATE: An adaptable framework to evaluate participatory research products

Background: Participatory research (PR) – the practice of involving stakeholders in research processes – is growing in prominence in health research because it can enhance the impact of research and the translation of research-based knowledge. Yet, the literature indicates that the products of PR studies are rarely evaluated empirically to track, demonstrate, and expand their claimed effectiveness and impact. This lack of measurement tools and frameworks can limit the effectiveness of knowledge translation (KT). Methods: We introduce a framework for evaluating the products of PR called PROLIFERATE. We use an inductive combination of formative and summative evaluation methods to pilot test the framework on a Frailty PR communication product (a video) to determine the methods’ functionality. Results: PROLIFERATE demonstrates adeptness for evaluating barriers and enablers of PR product uptake, effectiveness, and impact. It can identify ways to address barriers by assessing knowledge user perspectives on the comprehensibility of the product, emotional resonance, motivation to change, and future accessibility. Conclusions: PROLIFERATE can enable longitudinal and cross-sectional measurement of PR products in implementation and integrated KT efforts. It can evaluate and track the effectiveness and impact of different types of PR products in a situational responsive manner which compares users, platforms, and other factors in a replicable way.

"An umbrella term to include all partnered research, including community-based participatory research (CBPR), action research, participatory action research, participatory evaluation, community engagement and patient engagement" [5, p. 2].
Successful PR requires establishing connections across multiple sectors, including community, government, and education, to better create and implement products and tools generated using KT approaches [1]. PR products, developed for and with healthcare providers, decision-makers, industry, and patients, include policy briefs, clinical practice guidelines, education and awareness strategies (e.g. promotion campaigns and video communication), and technologies, such as digitized decision support systems and mHealth/eHealth applications [1,6]. While there are many sources of guidance on the development of PR and KT and implementation strategies [7,8], there are comparatively fewer on assessing their impacts [7,9,10].
The effectiveness and impact of PR products are rarely tested, empirically investigated, or systematically evaluated [7,9,10]. Evaluating how a PR product in uences its users considering different mechanisms of such interaction, and identifying ways to reduce any barrier that may be interfering with the product' intended in uence (co-designed objectives), are necessary steps to understand and enhance the effect of PR products [6,7,8,9,10]. This type of impact and effectiveness assessment could help to ascertain ways in which PR methods are positively in uencing health-related outcomes [7]. This lack of empirical evaluation procedures represents a gap between research and practice, which is constantly manifested in calls for evaluation frameworks and impact metrics connected to PR [1,2,3,4,5,6,7,8,9,10,11,12].
Thus, bridging this gap through the evaluation of PR products is necessary because billions of dollars are lost yearly in health research that fails to create a signi cant impact [7].
Design requirements for generating an inductive evaluation framework of PR products Evaluations of PR have been criticized for focusing on the short-term effects of their products and mainly using qualitative studies and results; their ndings are rarely generalizable or transferable because they concentrate on small samples or highly specialized topics [7]. Therefore, to address these gaps, frameworks for conducting cross-sectional and longitudinal evaluations of PR products should adhere to the design standards for public health evaluations [7,13,14,15].
Utility refers to the purpose of the product and how users will bene t from it [7,13]. Feasibility refers to how measurable is the product within each stage of its development/use and its practicality [7,13].
Propriety focuses on examining user experience by asking questions about the intended and unintended consequences of the product [13]. Accuracy relates to the success or failure of the product and its design while detecting and tracking such characteristics in a replicable manner across different populations, topics, and sectors [13,12,1]. Addressing these standards for generating an inductive evaluation of PR products requires a mixture of formative and summative evaluation methods [16].
Formative evaluation involves assessing process, experiential, or narrative aspects associated with the performance of the product [16]. These assessments are typically conducted using qualitative research methods (e.g., interviews) [16]. The summative evaluation considers the short-term to long-term effectiveness and impact of the product, primarily using quantitative measures [16]. A mixture of formative and summative evaluations embedded within frameworks designed to support productive partnerships to enact change and quality improvements while involving the community [17] may help to meet the utility and feasibility standards for the evaluation of PR products.
The Learning, Evaluation and Planning (LEAP) framework facilitates the mixture of formative and summative evaluations [17]. LEAP describes an iterative procedure for co-creating resources and service responses through participatory learning, evaluation, and planning [17]. The framework enables an assessment of needs, planning, implementation, empowerment, and review of actions, focusing on responding collaboratively to framing questions [17]. These questions could be used to guide the process of merging different evaluation tools to measure and track the impact of PR products.
Additionally, utility, feasibility and propriety standards can be met and tracked using a Body Mapping technique [14], which is an adaptable research tool that can capture baseline data, changes, and longitudinal feedback about a product or service from various types of knowledge users, including people with limited literacy or/and language di culties [14]. Body Maps typically depict a simpli ed gure of a human body with open-ended questions about the value, acceptance, utility, and comprehension of the product [14].
The combination of LEAP and Body Mapping can collect the perspective of knowledge users to assess the impact of PR products as well as identify their unintended consequences across time by conducting multiple measurements of the same product's effectiveness longitudinally [13,14]. Regarding the accuracy and propriety standards, the Net Promoter Score (NPS) [18], a metric tool from business research, can be used to evaluate products or services across large samples from a consumer-centric perspective [19,20]. The NPS calculates the number of respondents expressing positive views about a product or service ("Promoters"), minus those with negative views ("Detractors") while disregarding, within its score calculation, all neutral responses ("Passive") [19].
Each of the mentioned techniques and metrics seems to bring an inductive but relevant strategy to respond to current evaluation gaps in the measurement of PR products' effectiveness and impact.
Combining and recon guring them to create and test an operationalizable and replicable evaluation framework is therefore a necessary and an important contribution to improve health research implementation (knowledge and practice). Consequently, we are examining our inductive approach by codeveloping, recon guring, rede ning, and explaining such a procedure as an evaluation methodology, which is introduced and tested in this manuscript.
Methods PROLIFERATE refers to our adaPtable fRamework tO evaLuate dIFfErent pRoducts of pArTicipatory rEsearch. It integrates our inductive approach by combining two evaluation aspects: A 'course of action', designed for the exible co-detection of indicators of product effectiveness [17] and the 'evaluation focus', which brings a exible set of questions that helps to measure and tracks the performance and potential uptake of any PR products and its parts [13,14,18,19,20] (as per Figure 1).
We drew upon the mathematical model of communication or sender-message-receiver model (SMR) [21] to provide the theoretical foundation of PROLIFERATE. The combination of this structure with our inductive approach helps to conceptualize, interpret, analyze and summarize (as per table 5) possible bene ts, barriers, and potential limitations of the product while considering the non-linear, and interactive processes behind potential knowledge user' uptake and/or perception of the PR product [1,23,24].
SMR identi es how different mechanisms mutually interact and in uence each other recognizing the noise or barriers to their effective interaction (e.g. PR product intended impact vs knowledge-user perception of it) [22]. In doing so, it helps to detect how barriers might be addressed to optimize such impact, see Table 1 Page 6/21

EVALUATION COURSE OF ACTION
Step 1. Identifying outcome indicators.
The step is modelled on a case-by-case basis and encourages re ection on the objectives and needs that generate and drive the evaluation. The evaluation course of action is identi ed (including knowledgeusers as co-evaluators) when responding collectively to only ve of the seven questions of the LEAP framework [17], as exempli ed in Table 3.
Step 2. Collecting new evidence about the product.
This step connects to the EVALUATION FOCUS through a 'co-adapted' version of the Body Map tool [14,23]. In this context, 'co-adapting' means adapting these evaluation questions to the product and its endusers or knowledge users: Q1: Do knowledge users comprehend the PR product or its functionality (e.g., the function or utilization of a policy, a message, an app, or communicational piece)?
Q2: Does the product evoke an emotional response or resonate with knowledge users? Q3: What barriers or noise interferes within the uptake of the product? Q4: Is the product capable of motivating its planned effects (e.g., responses or intentions towards an individual behavior, education, communication, or system-level change)?
Q5: What are the potential avenues for future product optimization, promotion, usage, commercialization, or dissemination?
Step 3. Assessing value and meaning and benchmarking effectiveness.
This step involves a qualitative and quantitative assessment of the perceived utility, feasibility, propriety, and accuracy aspects of the PR product. It incorporates socio-demographic, institutional, historical, and other complex factors in uencing the PR product perception [23,24] (see Figure 1).

Assessing value and meaning
Evaluators collate responses for content analysis considering socio-demographics [25] and a preliminary coding structure is created. This initial coding is iteratively revised to ensure the accuracy of the response categories (i.e., treating participant groups as units of analysis). To help identify patterns, within-and between-group differences and similarities [25], numerical tabulations are calculated. Themes categories are then described, and rigor is promoted using an audit trail and in-person and virtual data analysis meetings in which ndings were cross-examined and veri ed.
For Q3 (noise) the coding requires [0 points] for neutral/positives responses (e.g., "I like everything about this product") and [1 point] for negative responses.
A Total Product Impact calculation involves interpreting all ndings and analyzing Q5 as positive [1 point] or [0 points] negative. This Total Product Impact calculation helps to identify the barriers (noise) within each component of the product (perceived internal noise, using negatives from Q1, Q2 and Q4) and any other external noise (Q3) about the whole product 'decodi cation', see Table 2: A researcher brie y introduced the video, before being projected on a large screen for the older sample and presented on a projector screen for the students (ML and RA, respectively). An information sheet was provided, and consent requested, as per Figure 3. Participants individually completed the body map questionnaire [14] which was co-adapted by our co-researchers: (PM)-a senior consumer, and (AM)-a nurse.
Age, gender, and postcode were captured by the body-map co-adaptation which presented a genderambiguous human gure and boxes placed symbolically as follows: Q1. Head: "This video made me think about …"; Q2. Chest: "From this video, I enjoyed this…"; Q3. Left hand (pointing to a waste bin): "Something from the video that didn't resonate with me was…"; Q4. Right hand (holding a bag): "Something to take away with me from this video…"; Q5. Beneath the gure: "This video is currently on YouTube, where else should it be available".
Step 3 of PROLIFERATE. Assessing value and meaning and benchmarking effectiveness This step is explained and presented as the result section.

Results
Assessing value and meaning for both study cohorts (PROLIFERATE Section 3.1) Theme 1: The animation was perceived as clear and informative. Overall, most older adults (66%) stated that they enjoyed the frailty animation while all the nursing students (100%) had positive impressions. They enjoyed the clarity and comprehensibility of the video (28%). Seniors found the animation informative and relevant concerning frailty prevention strategies and their current or future situation physical abilities ("it made me more aware of my body"; Female, 81 years old).
Five older respondents (17%) identi ed the animated video style as a novel attribute that facilitated understanding of frailty (e.g., "[the video] de ned the aspects of frailty pictorially, love the graphicscould help hearing and or sight-impaired to understand simplistically" (Female, 69 years). Similarly, most students (55%) found the animation clear and informative (e.g., "I enjoyed learning the basic concept of frailty. The content was enough and easily understandable") (Female, 20 years). Several students (27%) described the video as clear and "simple to understand" (Female, 30 years) with 18% explicitly stating that they enjoyed it.
Theme 2: Aspects of the video that did not resonate with older people. Almost half (48%) of them identi ed at least one aspect of the video that did not resonate with them. In comparison, none of the nursing students identi ed any speci c areas for improvement but noted that frailty also affects other age groups. Approximately one third (36%) of students described the video as relevant and useful (e.g., "everything discussed in the video was worth knowing about", Female, 35 years). Among older respondents, (31%) stated that the animation moved too quickly, particularly those with sensory impairments (e.g., "far too quick for older people, whom I assume are the ones it was produced for", Female, 73 years"). A minority of older respondents (7%) felt that the language was vague and questioned the accuracy of the information (7%). Theme 3: The animation increased awareness about the importance of addressing frailty. All of the students (100%) and a majority of older adults (69%) stated that the animation prompted them to consider aspects of frailty, including speci c strategies (e.g., diet, physical exercise) to prevent, delay, treat, and reverse frailty. One student re ected, "frailty is treatable if we recognize it in time and…take simple steps to reduce frailty" (Female, 24 years).
Over a third of the older adults (35%) mentioned that the video caused them to re ect on their circumstances and about a family member or friend (e.g., "[the video made me think about] all of the things that are really happening in my life", Female, 71 years) and ("…my own situation as my husband has recently spent time in [hospital] having a fall") (Female, 77 years). Similarly, 27% of students stated that the video made them think about their own and older adults' circumstances (e.g., "…my own health and my parents and ask them to use it", Male, 32 years). Additionally, students (18%) indicated that the animation caused them to re ect on negative aspects of frailty its misconceptions.
Theme 4: The video has the potential to in uence behavioral change intentions. Many older people (66%) agreed that they would "take something home" from the animation; (31%) did not respond to this question. (91%) of the students identi ed a positive take-home message. Nearly half (45%) of the older adults identi ed prevention/management strategies, including general health behaviors (e.g., "do something about being healthy", Male, 66 years) and speci c actions (e.g. "walking, muscle strengthening", Female, 69 years).
Most students (81%) identi ed the possibility of preventing frailty as a key take-home message, along with increased knowledge and awareness of frailty (64%). This included knowledge about "how frailty can be controlled and treated" (Female, 35 years). Some older respondents (17%) commented that they have increased their understanding of their own health and circumstances or someone they cared for (e.g. "my own situation as I have had several falls", Female 85 years).
Theme 5: Various opportunities exist to promote the video and its messages: (41%) of older adults and all students see future avenues for dissemination and promotion of the video. Of the older respondents who provided a recommendation, (24%) suggested screening the animation in a community setting, such as seniors' clubs (Male, 85 years), lifestyle villages (Male, 77 years), and retirement villages (Female, 77 years). For example, one participant commented, "many older people do not have access to Internet or only know basic things (…) so perhaps having it shown by community speakers, through councils' programs" (Female, 68 years). Other suggestions among older people were screening the video in a healthcare setting (e.g., waiting rooms in general practice surgeries, dentists, and outpatient clinics; 10%), print and broadcast media, such as television (7%), and online media, such as healthcare and/or community organization websites (7%).
Many students (73%) suggested online media, including e-libraries and e-books, social media, government and university websites, and virtual learning platforms such as Blackboard as channels to share the video. They also suggested screening the animation in healthcare settings (36%), such as hospitals and "different local and rural health centers" (Male, 22 years) and aged care facilities. Other dissemination methods suggested by students included traditional media (e.g., national television, newspapers; 18%), screening in the community (e.g., public libraries, churches; Male, 22 years), and distributing the animation to relevant organizations (18%).
Benchmarking effectiveness (PROLIFERATE Section 3.2) Quantifying product effectiveness for Cohort 1, as per Table 5 The PR product met the benchmarking criteria [(>0)] for Cohort 1 by receiving several ratings above zero in all its components. The highest performing aspect was emotional resonance (7 points). This was followed by comprehension and motivation to behavioral change (5 points).
Quantifying product effectiveness for Cohort 2, as per Table 6 The benchmarking criteria were met in all the aspect of the PR product. The highest-rated aspects were comprehension and emotional resonance (19 points each, respectively), followed by the motivation to behavioral change (17 points). A summary of the total product impact considering the formative and summative evaluation is presented in Table 7.

Discussion
PROLIFERATE appraised PR product' effectiveness and impact on two cohorts via a quick process which requires a short and simple data collection and analysis. The method can be employed to track product impact over time if multiple measurements are taken with the same or similar population. It can detect possible mechanisms behind product performance (how and why). Creating and testing PROLIFERATE required of an explicit focus on evaluating the effectiveness (enablers/barriers) of PR product impact, via an iterative and inductive participatory approach.
PROLIFERATE shows that the video achieved its intended impact as both audiences demonstrate excellent emotional resonance with the material (Q2), comprehended the messages (Q1), and indicated that were motivated to apply preventive strategies concerning frailty (Q4). The barriers concerning product uptake (Q3/Noise) were identi ed and did not affect product impact (Σ Benchmark-PRO). Insights (from Q3 and Q5) can inform technical and semantic optimization strategies to cement knowledge and increase message reach (e.g. 'semantic barriers' observed in nursing students possibly explained by the in uence of institutional and socio-cultural differences [32,33,34] demonstrate the acuity of the approach).
Limitations: The framework was not informed by a systematic review or a targeted literature search to identify/compare equivalent frameworks, and our team did not seek other experts input for its design. Therefore, it may not have captured all the dimensions relevant evaluating different types of PR products.
To compensate for these limitations, we empirically tested our inductive approach to provide evidence of its originality and demonstrate its adeptness. Future studies need to automatize PROLIFERATE to improve its multiphasic utilization considering large samples. Such automatization may present important challenges: (i) embedding PROLIFERATE into a technological tool, (ii) considering health status variables as mediators of uptake/impact, (iii) modelling impact (e.g. [35,36,37]).

Conclusion
PROLIFERATE introduces a consistent, standardized, way to obtain and analyze data about PR products, facilitating a uniformed quanti cation of the product and its parts, and informing the delivery of targeted Depicting PROLIERATE's evaluation cycle, including the course of action and its focus components. Modi ed body map tool.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. RevisedCHEERSChecklist.docx