PROLIFERATE refers to our adaPtable fRamework tO evaLuate dIFfErent pRoducts of pArTicipatory rEsearch. It integrates our inductive approach by combining two evaluation aspects: A ‘course of action’, designed for the flexible co-detection of indicators of product effectiveness [17] and the ‘evaluation focus’, which brings a flexible set of questions that helps to measure and tracks the performance and potential uptake of any PR products and its parts [13, 14, 18, 19, 20] (as per Figure 1).
We drew upon the mathematical model of communication or sender-message-receiver model (SMR) [21] to provide the theoretical foundation of PROLIFERATE. The combination of this structure with our inductive approach helps to conceptualize, interpret, analyze and summarize (as per table 5) possible benefits, barriers, and potential limitations of the product while considering the non-linear, and interactive processes behind potential knowledge user’ uptake and/or perception of the PR product [1, 23, 24].
SMR identifies how different mechanisms mutually interact and influence each other recognizing the noise or barriers to their effective interaction (e.g. PR product intended impact vs knowledge-user perception of it) [22]. In doing so, it helps to detect how barriers might be addressed to optimize such impact, see Table 1.
Table 1. The theoretical basis of PROLIFERATE drawing from SMR model.
EVALUATION COURSE OF ACTION
Step 1. Identifying outcome indicators.
The step is modelled on a case-by-case basis and encourages reflection on the objectives and needs that generate and drive the evaluation. The evaluation course of action is identified (including knowledge-users as co-evaluators) when responding collectively to only five of the seven questions of the LEAP framework [17], as exemplified in Table 3.
Step 2. Collecting new evidence about the product.
This step connects to the EVALUATION FOCUS through a ‘co-adapted’ version of the Body Map tool [14, 23]. In this context, ‘co-adapting’ means adapting these evaluation questions to the product and its end-users or knowledge users:
Q1: Do knowledge users comprehend the PR product or its functionality (e.g., the function or utilization of a policy, a message, an app, or communicational piece)?
Q2: Does the product evoke an emotional response or resonate with knowledge users?
Q3: What barriers or noise interferes within the uptake of the product?
Q4: Is the product capable of motivating its planned effects (e.g., responses or intentions towards an individual behavior, education, communication, or system-level change)?
Q5: What are the potential avenues for future product optimization, promotion, usage, commercialization, or dissemination?
Step 3. Assessing value and meaning and benchmarking effectiveness.
This step involves a qualitative and quantitative assessment of the perceived utility, feasibility, propriety, and accuracy aspects of the PR product. It incorporates socio-demographic, institutional, historical, and other complex factors influencing the PR product perception [23, 24] (see Figure 1).
3.1 Assessing value and meaning
Evaluators collate responses for content analysis considering socio-demographics [25] and a preliminary coding structure is created. This initial coding is iteratively revised to ensure the accuracy of the response categories (i.e., treating participant groups as units of analysis). To help identify patterns, within- and between-group differences and similarities [25], numerical tabulations are calculated. Themes categories are then described, and rigor is promoted using an audit trail and in-person and virtual data analysis meetings in which findings were cross-examined and verified.
3.2 Benchmarking effectiveness
Evaluators quantify positives, negatives, and neutral values considering NPS’ basic principles [18]. For Q1, Q2, Q4 the coding requires [2 points] for positive responses (e.g., Q1 response: “I really understand this product”) and [1 point] for negative responses (e.g., “I don’t understand this product”); and [0 points] for neutral responses (e.g., blank or unclassifiable responses).
For Q3 (noise) the coding requires [0 points] for neutral/positives responses (e.g., “I like everything about this product”) and [1 point] for negative responses.
A Total Product Impact calculation involves interpreting all findings and analyzing Q5 as positive [1 point] or [0 points] negative. This Total Product Impact calculation helps to identify the barriers (noise) within each component of the product (perceived internal noise, using negatives from Q1, Q2 and Q4) and any other external noise (Q3) about the whole product ‘decodification’, see Table 2:
Table 2. PROLIFERATE formulas
The function of PROLIFERATE formulas can be appreciated in Figure 1, which describes the evaluation framework and its cycle.
Pilot testing PROLIFERATE
We pilot tested PROLIFERATE on a 2 minute 51 second animated video, co-developed in Australia by the National Health and Medical Research Council (NHMRC) funded Centre of Research Excellence in Frailty Research and Healthy Ageing, see Figure 2. This information is expanded next in Table 3 by applying PROLIFERATE first step.
Step 1 of PROLIFERATE: Identifying outcome indicators.
Table 3. Outcome indicators responding to five LEAP questions.
Step 2 of PROLIFERATE: Collecting new evidence about the product.
This data collection was approved by the Social and Behavioral Research Ethics Committee of Flinders University (Project No. 8474). We used purposive and convenience sampling as per Table 4.
Table 4. Sampling
A researcher briefly introduced the video, before being projected on a large screen for the older sample and presented on a projector screen for the students (ML and RA, respectively). An information sheet was provided, and consent requested, as per Figure 3. Participants individually completed the body map questionnaire [14] which was co-adapted by our co-researchers: (PM)- a senior consumer, and (AM)- a nurse.
Age, gender, and postcode were captured by the body-map co-adaptation which presented a gender-ambiguous human figure and boxes placed symbolically as follows: Q1. Head: “This video made me think about …”; Q2. Chest: “From this video, I enjoyed this…”; Q3. Left hand (pointing to a waste bin): “Something from the video that didn't resonate with me was…”; Q4. Right hand (holding a bag): “Something to take away with me from this video…”; Q5. Beneath the figure: “This video is currently on YouTube, where else should it be available”.
Step 3 of PROLIFERATE. Assessing value and meaning and benchmarking effectiveness
This step is explained and presented as the result section.