We have previously published the study’s protocol in CMAJ Open (12). Below is the methods summary of the vaccine recipients’ arm of the study. We used the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) Short Form (SF) (Additional file 1) (13) as well as the Realist And Meta-narrative Evidence Syntheses: Evolving Standards II (RAMESES II) (Additional file 2) to report methods and data analysis (14).
Setting
Our interdisciplinary research team was based in Saskatchewan and consisted of three PFPs (C.S., B.A., G.F.), five realist evaluators (A.R.A., T.C., N.M., T.V., G.G.), two Saskatchewan Health Authority employees (A.R.A., J.V.), one Saskatchewan Health Authority (SHA) policymaker (C.H.) and one research assistant (M.Y.). The study was conducted virtually due to COVID-19 restrictions.
Our study focused on the cities in which the pilot phase of Saskatchewan’s COVID-19 vaccination program occurred (i.e., Regina, Saskatoon, and Prince Albert) (15, 16). The Government of Saskatchewan and the SHA used pilot term to describe administering vaccine to high-risk and vulnerable population (17). The first COVID-19 vaccination was delivered in Regina (December 15, 2020), followed by Saskatoon (December 22, 2020) and Prince Albert (January 7, 2021). The pilot phase ended in March 2021 when COVID-19 vaccines were administered to high-risk populations of priority healthcare workers, elderly residents, and northern communities (17–19).
Study Design
We used realist evaluation, a theory-driven approach where researchers examine the question of “what works, for whom, in what circumstances, how and why?” through the development of a program theory (20–23). Typically, realist evaluation encompasses three phases: 1) eliciting the initial program theory(s), 2) testing the initial program theory(s) through data collection, and 3) formulating a refined final program theory(s) based on the findings of collected data in phase two (24–27). In developing, testing, and refining a program theory, realist evaluators assume that an outcome (vaccine uptake, in this study) is generated by mechanisms being triggered in specific contexts through an actor(s) (vaccine recipients, in this study) (27). These causal relations, referred to as CMO configurations (CMOCs), are the building blocks of program theory (28).
In phase one of this study (January – May 2021), the PFPs received training about realist evaluation by A.R.A and T.C (explained in more details under the Patient Engagement section). The initial program theories (IPTs) were co-developed with PFPs to identify the vaccine program’s underlying assumptions, outcomes of interest, and proposed mechanisms of achieving the targeted outcome (12). We reviewed COVID-19 vaccination peer-reviewed and grey literature as well as COVID-19 vaccination program documents and communications (e.g., SHA implementation meeting notes) from the three sites (29). Three categories of IPTs were developed (Fig. 1, Fig. 2, and Fig. 3), one for each group of actors (ICU/ED physicians and healthcare workers; LTC managers and healthcare workers; and LTC residents) in the pilot phase. The identified mechanisms for the recipients were overlapping (Fig. 4), hence, we grouped them into “mechanism chain” with two categories: reasoning and cognitive/emotional mechanisms (24). In phase two of the study (June – September 2021), we tested the IPTs through online interviews. In phase three (October – December 2021), we refined and validated the IPTs by retroductively analyzing the interview data and generated the final program theories for vaccine recipients.
Patient Engagement
We used the Saskatchewan Centre for Patient Oriented Research (SCPOR) Patient-Oriented Research Level of Engagement Tool (PORLET) (30) to facilitate and guide the level of engagement with the PFPs. We selected the three PFPs purposefully due to their prior experience and engagement in a variety of health research studies including patient-oriented realist research. Their background brought perspectives of social workers, health educators, cancer survivors, and family members and caregivers of LTC residents to the team. We held 16 meetings with PFPs: nine meetings to train the PFPs using the training materials for realist evaluation (31), as well as discuss and develop IPTs, a participants’ invitation letter and consent form, an interview guide, a plain language version of the IPTs, and recruitment strategies; three meetings to run mock-up interviews; three meetings to review preliminary analysis results; and one final research team meeting to finalize the final program theories.
A.R.A. and T.C. used Mural boards (online platform) to train PFPs on realist evaluation program theory development. The boards contained tables with context, mechanism, and outcome columns for each group of vaccine recipients. The tables were prepopulated with literature samples related to vaccination. During the training sessions, A.R.A. and T.C. demonstrated how to extract contexts, mechanisms, and outcomes from the samples. Subsequently, further sample resources were co-analyzed by the PFPs. This approach led to build the IPTs in collaboration with the PFPs.
Along with A.R.A., PFPs were each assigned to two interviews based on their previous research experience and background. PFPs are engaged in knowledge translation such as dissemination of findings at conferences or a manuscript format.
Participants
The study population was composed of all individuals in Saskatchewan who were eligible for a COVID-19 vaccine during the pilot phase, including healthcare workers in intensive care units / emergency departments / COVID-19 units, staff at testing and assessment centres, elderly residents in long term care (LTC), older adults over 80 years of age, and residents in remote northern communities. Caregivers of LTCresidents were also eligible for inclusion when residents were not physically or mentally able to participate in an in-depth interview. We recruited participants through PFPs and the Saskatchewan Care Network, using purposive sampling to ensure diverse inclusion of healthcare and non-healthcare workers.
Data Collection
To test the IPTs developed in phase one, a PFP and A.R.A. co-conducted one interview with each participant (30–45 minutes). The three PFPs and other members of the research team co-developed an interview guide based on a realist evaluation teacher-learner approach (Additional file 3) (32). The PFPs and researchers also co-created a plain language version of the IPTs (Additional file 4). After receiving consent from the participants, interviews were audio-recorded and later transcribed. Participants were offered an honorarium for their participation.
Data Analysis
Qualitative data collected from interviews was analyzed using a retroductive approach common in realist research (21–26). As such, we employed both inductive and deductive coding (21) and used the team’s insights to understand what caused vaccine uptake (refer to the protocol paper for detailed information on data analysis) (12). Two researchers (A.R.A., T.C.) conducted the preliminary analysis to identify the CMOCs from each interview. They compared and contrasted the CMOCs with the IPTs and refined the IPTs accordingly. The findings of the preliminary analysis were then reviewed by each PFP who co-conducted the corresponding interview. All findings were discussed in a final meeting where all research team members were present.
Ethics Approval
The study was part of a larger project “Developing a patient-oriented realist evaluation for COVID-19 vaccine implementation in Saskatchewan” which received a letter of exemption from the University of Saskatchewan Behavioural Research Ethics Board and the Saskatchewan Health Authority Research Ethics Board because of its program evaluation status (33). The study’s consent forms reflected this exemption status.