This field test will use a multiple method qualitative design. It will include training of participants, standardized data collection using desk-based templates, and follow-up qualitative interviews with both the assessors and those whose projects have been assessed. As well, it will include a consultative process with the project team for revising RQ+ 4 Co-Pro based on the outcomes of the field test [35,36].
The study will take a research co-production approach. To do so, the study is being undertaken as a partnership between researchers and knowledge users. All activities and responsibilities will be shared, yet, five team members (authors: RKDM, FC, IDG, AK, CM) are primarily responsible for study design and execution. Thirteen team members (authors: ABA, RA, JB, CEC, OD, EDR, LAF, MG, AMH, RK, SK, JR, GS) hold primary responsibility for identifying knowledge uptake and use opportunities. These ‘knowledge user’ team members represent critical stewardship roles for research co-production broadly, including: funders, university administrators/leaders, research evaluation specialists, journal editors, co-production trainees, research managers, and co-production scientists. By working together to field test RQ+ 4 Co-Pro we hope to spark reasoned and appropriate uptake of the framework into settings where current co-production evaluation techniques demand revision and innovation.
The field test will be implemented in four phases, which comprise eight steps. Figure 3 presents an illustration of the complete research life cycle.
PHASE 1 – STUDY PREPARATION
1. Sample recruitment and participant consent
Researchers in the Integrated Knowledge Translation Research Network (IKTRN) will be invited to submit projects for assessment and volunteer to assess another project. IKTRN is a network of researchers with an interest in both using and carrying out research on integrated knowledge translation. IKTRN is funded by a multi-year grant from the Canadian Institutes of Health Research [19]. The sample will be a convenience sample. This sample will be drawn from researchers with IKT research experience (members of the IKTRN) and who have recently completed an IKT project (have an IKT case published in the IKTRN casebook series). This invitation will be delivered by email from the study PI to eligible members of the IKTRN, until the desired sample size of 16 projects is reached, with a maximum of 20 projects. Enrolled participants will be arranged in dyads based on research topic familiarity for the assessment.
The sample range (16-20 projects) is based on two factors. The first is viability given resource requirements of past experiences using the RQ+ approach, and our own study timelines and resources for this project. The second is the anticipated saturation point of qualitative data collected in the field test [37].
1.1 Eligibility criteria
We will consider eligibility at two levels: (1) the IKT research project and (2) the individuals participating in our study, as described below in tables 4 & 5 respectively. The research team will gather informed written consent from all participants.
2. RQ+ 4 Co-Pro framework training
Participants will receive training in RQ+ 4 Co-Pro. Training will be provided by the core research team, with the aim to introduce the RQ+ 4 Co-Pro framework, the definitions and meaning of its components (contextual factors, quality dimensions and subdimensions, evaluative rubrics) and systematize the approach to its use by participants. The 2-hour training will be completed prior to the initiation of all data collection.
PHASE 2 - DATA COLLECTION
Data collection will involve four steps: (1) completion of a participant socio-demographic form, (2) completion of a project information form, (3) dyadic RQ+ 4 Co-Pro assessments, and (4) participating in an interview with the research team on the strengths and limitations of the RQ+ 4 Co-Pro framework.
Step 1 - Participant socio-demographic form
All participants will be sent a link to an online socio-demographic form. This form will collect information on participant demographics and their experience/background developing and/or delivering IKT research projects. We will ask that participants complete this form prior to taking part in the training session. (5 minutes)
Step 2 - Project information form
All participants will be sent a link to a project information form. This form will be pre-populated by the study team as much as possible to profile the project included in the field test. The study participant will verify and complete any missing information on the project profile prepared by the study team. (10 minutes)
Step 3 – Dyadic project assessment exercise
Each participant in the paired dyad will provide the other, who will serve as assessor, with publicly available documentation on the project they will be assessing (inter alia publications, manuscripts, reports, briefs, blogs, etc). These assessors (study participants in dyads [38]) will review this material to gain an understanding of the project context, as well as its strengths and weaknesses (1-2 hours). Next, the assessors within each dyad will engage in an assessment interview about their projects using the RQ+ 4 Co-Pro training and the field test template (see Annex 1) provided by the research team. Assessment interviews may be done in one virtual call or split in two as the two determine. It is estimated they will last 60 minutes per project. The field test template will be used by the assessors for recording results of the assessment during the interview.
Step 4 – Research interview with RQ+ 4 Co-Pro study team
On completion of the dyadic assessments, members of the research team (RKDM, FC) will interview the assessors (study participants) individually using a semi-structured interview guide to elicit their views as both assessor and assessed on the utility and relevance of using RQ+ 4 Co-Pro to assess the quality of IKT research [36]. The interviews will be completed by phone or video conference, depending on participant preference. Interviews will take approximately 60 minutes.
PHASE 3 - ANALYSIS AND REVISION
1. Data Analysis
Data analysis will be conducted for each data source independently (Participant demographic forms, Project information forms, Interviews with study participants), and triangulation will be conducted across the independent lines of evidence for congruence as well as instances of discordance.
Step 1 – Participant demographic form analysis
Frequencies will be generated for all closed-ended questions. Responses to open-ended questions will be analyzed for common and disparate themes using content analysis. Analysis will provide an overview of participants backgrounds and experiences brought to the field test.
Step 2 - Project information form analysis
The project profile forms will be analysed using content analysis to provide an overview of the nature of the projects included in the field test.
Step 3 - Interview analysis
Qualitative interview data collection and analysis will occur simultaneously so that identified themes can be incorporated into future interviews. Interviews will be audio recorded with permission of the interviewee. Where permission for transcription or recording are not granted the interview notes will be sent to the interviewee for review.
We will use thematic analysis [39] to identify patterns in the interview data. We will use an inductive or data-driven approach, without using a pre-existing coding frame. The coding will be modified based on new findings and in collaboration amongst interviewers. The first two interviews will be coded by two researchers independently and the results compared. Differences will be discussed to ensure agreement on a common approach for the remaining interviews. If agreement is not achieved between the two researchers, a third researcher will arbitrate opposing views and provide a third opinion to reach majority decision if consensus is not achieved.
Step 4 – Triangulation and analysis
As a final step in data analysis, we will look for similarities and differences of note in the study data by comparing findings across the lines of evidence. We will conduct triangulation by data source and by data collection method. Data will be considered in triangulation by using identified codes and themes to compare data. For example, we may cross tabulate all projects with a timeline of more than four years (as identified in the project information form), by perspectives around the importance, or lack-thereof, of using contextual factors in the RQ+ 4 Co-Pro framework. This is a hypothetical example. Triangulation will be driven by identified themes in the data.
2. Revision of the RQ+ 4 Co-Pro Framework
Based on the findings of the research, we will revise the prototype version of the RQ+ 4 Co-Pro framework. To facilitate this revision, the research team will host a meeting of all team members (including our knowledge users) to review and discuss preliminary research results and how these may induce the desire for change to the framework or its components. The reasons for changing a framework component will relate to the two research questions driving the study: relevance of the framework components and utility of the framework components and its application. Following team iteration, we will prepare any required revisions, and represent the revised framework to the study participants for review/member checking.
PHASE 4 – RESULTS SHARING
The final version of the RQ+ 4 Co-Pro framework will be published in a study findings report. This report will be submitted to an open access peer-reviewed journal for external assessment by co-production specialists. Uptake and use strategies will be developed by/amongst knowledge user perspectives represented on our co-production team.