Following a co-production approach, the research was undertaken by a team of researchers and research beneficiaries from a broad range of professional backgrounds. All members participated through the lifespan of the research process, collaborating to develop study objectives, questions, methodological design and results sharing strategies. This collaboration was conducted using virtual MS Teams meetings and email-based document exchange. Although the project employed a collaborative approach to all steps, within the team authors RKDM, FC, and IDG were primarily responsible for oversight of research implementation and led activities such as data collection and academic reporting. Authors ABA, RA, JB, CEC, OD, EDR, LAF, MG, AMH, RK, AK, SK, CM, JR, GS were primarily responsible for identifying and actioning knowledge uptake and use opportunities. Purposefully, the co-production team was constructed to represent critical stakeholder and gate-keeper authorities for research co-production, including: co-production scientists, journal editors, university leadership and administration, graduate students, funders, research network management, research evaluators, public policy makers, health systems consultants, and not-for-profit/foundation senior leadership. The co-production team included members based in Australia, Canada, and the United Kingdom. Members of the team self-reported gender, and the aggregate profile is 13 women (72%), 5 men (28%) and 0 non-binary (0%).
Research design
We used a constructivist paradigm (wherein those involved construct their own knowledge of the world through experience and reflection) and employed a multiple methods qualitative design (14;15). The constructivist approach facilitated co-production across perspectives on our team, by centering the importance of deliberating varied experience and interpretation in our practice and research process (16). The constructivist approach supported qualitative data collection by elevating participants’ conceptual views about and experiences with the RQ + 4 Co-Pro field-test, and the paradigm of constructivism drove our use of an inductive approach to data analysis (17;16). We used standardized self-reported participant and project templates, training of participants, participant-led dyadic evaluations of projects, and follow-up qualitative interviews with participants (who were all both assessors and those whose projects had been assessed). To develop and revise the RQ + 4 Co-Pro Framework, the co-production team employed a process of collective examination of empirical results and deliberative dialogue (18;19;20). This process allowed the team to identify possible revisions and select desirable changes by a process of creating consensus (16). Figure 1 illustrates the life cycle of the research.
<<Insert Fig. 1>> – Research life cycle
Phase 1 – Study preparation
1.1 Sampling
Individual participants and individual projects were purposefully drawn from the Integrated Knowledge Translation Research Network (IKTRN). The IKTRN is an international network of researchers and knowledge users who practise and study integrated knowledge translation (21). We identified IKTRN as the sample universe on two grounds. First, to ensure study participants were suitably skilled and experienced in co-production. Second, members of the IKTRN would be able to submit a recent co-production project of their own for dyadic evaluation. Sixteen to 20 participants were identified by our co-production team as an estimated sample size that would lead to saturation (22;2). We were open to increasing the sample size if the saturation estimate did not hold (in terms of data richness). As discussed under the interview analysis section, this original sample size estimate in fact did hold. The IKTRN Director recruited network members through email correspondence. Once a sample of 20 individuals was recruited, we accepted all 20 participants and obtained informed consent for study participation. Recruited individuals were asked to identify a recent IKT study with which they were involved. No other limitations were placed on the identified study. During the study, 2 participants dropped-out due to competing work demands, leaving 18 active participants.
1.2 RQ + 4 Co-Pro training of participants
We hosted an online training session that introduced study participants to the RQ + approach and the novel RQ + 4 Co-Pro Framework we developed in prototype version for this study. The training session was two hours in length. It was led by members of the co-production team with extensive experience using the RQ + approach at the International Development Research Centre and at the Global Challenges Research Fund (RKDM & FC).
Phase 2 – Data collection
2.1 Participant & project information forms
Each study participant completed two digital forms. The participant information form collected basic demographic details about the participant, including gender, years of experience with co-production, and years of experience in research. The project information form elicited basic information about the identified project the participant would represent in the evaluation simulation, such as project length, funding amount, and type of knowledge-users involved. Collecting systematic data about each participant and project allowed us to examine and better understand the evaluators and evaluands (subjects of evaluation), that made-up the field-test.
2.2 RQ + 4 Co-Pro evaluation field-test
Following training and basic information collection, study participants were randomly assigned into dyadic sets. We launched with ten dyads, but as noted earlier, one pair dropped out of the study leaving 18 participants in nine dyads. Dyads were the core structure of the RQ + 4 Co-Pro field-test (23). Dyads exchanged project background publications and scheduled their own interviews. Using data collected from project documents and the interview, each participant used the RQ + 4 Co-Pro Assessment Instrument to evaluate their partner’s project. The prototype Assessment Instrument used by study participants was published as Additional File 1 in our study protocol; see: McLean et al. 2022a (2). The field-tested version, including updates and changes driven by this research, is described in the following section of this paper and published as Additional File 1. Completing the field-test required participants to evaluate their dyadic partner’s project that included all three tenets of the RQ + Approach: 1) considering context, 2) reviewing and assessing multiple dimensions of quality, 3) use of an empirical and systematic approach that incorporated a variety of data sources to triangulate findings. Dyads did not return their completed RQ + 4 Co-Pro Assessment Instruments to the research team as the purpose of our research was not to assess the quality of the sampled projects but to test the relevance and utility of the Framework.
2.3 Qualitative interviews with study participants
To learn about participants’ experiences using the RQ + 4 Co-Pro Framework and Assessment Instrument in the field-test, we interviewed each study participant independently for 45–60 minutes. We elected to use a qualitative approach to data collection to capture the context, diversity and richness of experience within the participant sample. Each qualitative interview used a common guide but was approached in a semi-structured manner to capture the feedback each independent interviewee found to be most pertinent to their experience. Interviewers (RKDM, FC) conducted the first interview together to ensure consistency of the approach and debriefed on the experience to discern possible improvements. Thereafter, interviews were conducted independently. The interviewers exchanged notes as interviews were completed to enhance coherence in the approach and ensure emergent learning was built into both interviewers’ work. Interviews were conducted using the MS Teams platform and lasted between 45 and 90 mins. Interviews were audio recorded and transcription was completed by the interviewer. Alternatively, with permission of the interviewee, transcriptions were generated within the MS Teams platform in real time and the interviewer took notes concurrently to censure a complete record of the interview was captured.
Phase 3 – Analysis & iteration
3.1 Data analysis
All data for each participant (transcripts and notes from interviews, reflections with participants, project information forms, participant information forms) were assigned a random number identifier for confidentiality purposes.
Project and participant information forms were analyzed using frequencies for close-ended questions. Open-ended questions were reviewed for common or disparate themes (24).
To analyze interview data, we used thematic analysis to identify patterns in the interview data (25). We used an inductive, or data-driven approach, without using a pre-existing coding frame. As interviews were completed, we met to discuss emerging themes and experiences to iteratively develop a data coding structure. The coding frame for the interviews was agreed/completed in a team meeting at the conclusion of all interviews.
We analysed data from each method (project information forms, participant information forms, qualitative interviews) separately using the above-described processes. Following independent analysis, we conducted triangulation across methods to identify patterns in the data. Our triangulation process was done through stratification of interview data by response categories in the project and participant information forms. We conducted stratified analysis for grant length, funder type, and participant experience teaching and supervising co-production. However, we identified no dominant patterns in the data when interview findings were stratified by project or participant characteristics. That is to say, the analysis produced comparable results under each stratified analysis. Consequently, we report study results in aggregate and by source/method in the following section of this paper.
3.2 Framework and Assessment Instrument iteration
Based on the data analysis, we revised the prototype RQ + 4 Co-Pro Framework and Assessment Instrument using deliberative discussion as a co-production team (18;19;20). To facilitate the deliberative revision process, we held two meetings of the co-production team where we discussed findings and recommendations derived from the interviews. Changes were agreed to by consensus at each meeting. In addition to team meetings, we coordinated deliberation and revisions via email; the final iteration was approved by all team members and is presented as Fig. 2 (the RQ + 4 Co-Pro Framework) and Additional File 1 (the RQ + 4 Co-Pro Assessment Instrument). In alignment with our research questions, we used two criteria to identify possible changes: 1) relevance of RQ + 4 Co-Pro for research co-production, and, 2) utility of RQ + 4 Co-Pro for co-production evaluation.