In this section, we present the five different steps taken to develop the Pan-Canadian PE in PS survey tool. Please refer to figure 01 for the summary of the different phase to build the survey tool.
Step 1: Literature review and revision of the initial tool
Methods: We performed a literature review to complement the work performed developing the French questionnaire and to capture publications published from 2017 to 2019 regarding best practices in PS and PE and/or evaluation tools to survey PE on PS. Health and social sciences databases (PubMed, Medline, Cochrane, CINAHL, EMBASE) were consulted using the keywords “tool” OR “assessment” OR “measurement” OR “questionnaire” AND “patient engagement” OR “patient participation” OR “patient involvement” OR “patient safety.” We also integrated comments and suggested modifications collected during the final validation phase of the French tool, conducted in the province of Quebec, Canada [33] and from a study conducted in France [36] (see tables 1 and 2 for more information).
Findings: The literature review highlighted four tools for evaluating PE and/or PS at different system levels (see table 1). Among the four tools identified, none tackled both PE and PS at the organizational or system level. The first tool focused on PE in PS at the clinical level, the second tool focused on PE only at the organizational level. The third and fourth tools focused on PE in research methods and its evaluation.
Suggested modifications from previous studies conducted in Quebec and France were also analyzed and integrated into the assessment of the creation of the pan-Canadian PE in PS survey tool (see table 2 for detailed information). In summary, participants in the Quebec and French studies reported that they generally had a good understanding of the tool and its questions as well as the instructions and the choice of the participating team in answering the questions :
“The tool helps us really frame our strategies and could be used as a self-assessment tool for healthcare institutions […] It will be used for sure.”
“You're asking very pertinent questions, and that helps me.”
“The choice of having a team answer the questionnaire is very important […] It has to be a group that answers the questions. No individual has a comprehensive view of what is going on in the institution.”
At the end of the evaluation of the Quebec and French studies, nine questions were deleted to avoid repetition and the rewording of certain items, and some questions were scaled to improve understanding. Questions 67-73 were repetitions so they were deleted (N= seven repeated questions deleted); two questions pertaining to Family Medicine Groups were also deleted, as they were confusing since not all healthcare organizations did not have Family Medicine Groups and since these questions were specific to the context of the province of Quebec. Other limitations of the previous questionnaire were about the percentages in the questionnaire that were difficult to complete since healthcare organizations did not necessarily know the percentage of engagement of their users in different instances and committees. We thus, modified these sections not to include the percentages, but to just let the participants answer about the degree of involvement of their users (ex: I don’t know, never, sometimes, always, in progress).
Another limitation were about questions on policies and procedures which needed not to just include if the organization had a certain policy or procedure, but also how these policies were integrated and experienced in the organization.
“The absence of policies and procedures does not reflect dedication. Pay attention to the presence of procedures and policies and how they are experienced.”
Modifications were brought to these questions (N= 25 questions on policies and procedures) and clarification were added to the main question. Here is one example :
Q. 32. The organization has a formal policy on disclosure
|
32a. The disclosure policy is periodically evaluated
|
32b. The disclosure policy is evaluated in partnership with patient advisors?
|
Step 2: Translation of the French tool into an English version
Methods: The French tool was translated into English and validated following the methodology (back translation) proposed by the Agency for Healthcare Research and Quality [37] and the methodology recommended by the United States Bureau of the Census [38] (see figure 2 for more information on the four steps used in the back translation).
Findings: The back translation was done in four steps:
- Translation of the French tool into an English version by the research team.
- Translation of the English version back into the original language by an external resource
- Comparing that new translation with the original French tool
- And, reconciling any meaningful differences between the two versions and bringing about the corrections.
Thus, the tool was translated from the French version used in the Quebec study [33] into an English version, and the English version was translated back into French to confirm the terms used. During the translation and the back-translation, we made sure not to change or modify the meaning or depth of the items in order to not affect the validity of the content.
Step 3: Creation of a Canadian expert advisory group
Methods: To adapt the French version, a Canadian expert advisory group was created to adapt the tool to suit a pan-Canadian context, considering the specific features of Canadian health systems. The criteria used to select members were: their knowledge of and expertise in the Canadian health system, patient engagement and patient safety; having already collaborated with CPSI, and their province of origin (for purposes of representation). An initial list of experts was established by CPSI and the research team, emails were sent to the identified individuals, and a final selection was made by the researchers and CPSI using the selection criteria.
Findings: In total, eight members from five Canadian provinces were recruited: 1 academic, 1 patient representative, 4 quality improvement and patient safety specialists, and 2 experts from Accreditation Canada. The research team in charge of developing the study objectives and methodology had five members (three PhD students, one researcher expert in PE, and one advisor from CPSI as a PS and PE projects expert) (see Table 3). In total the working group had 13 members, met 16 times (the research group met with the expert advisory group 4 times and 12 times alone), and on average the meetings lasted 2-3 hours.
Step 4: Assessment, adaptation and editing
Step 4.1: Assessment and adaptation of the tool content
Methods: The Canadian expert advisory group reviewed the initial translated English tool for assessment and adaptation purposes. The expert advisory group were instructed in their revision of the tool to focus on each question and on a more general aspect of it by keeping in mind the flowing interrogations :
- Wording of the questionnaire?
- Themes covered?
- What is missing or redundant?
- Relevance of the questions?
- Target population (who can answer the questionnaire)?
- Can the questionnaire be used to assess the reality of patient safety in Canada (so that CPSI and the provinces can use the findings to make better decisions and update the PE in PS guide)?
The goal of the initial tool review and assessment by the expert advisory group was to work as a group using a consensus-building approach to evaluate every single dimension of the initial questionnaire, in order to arrive at a consensus on the content validity of the entire tool (i.e. so that all the dimensions used in the Canadian survey would be as relevant, reliable, and exhaustive as possible to fully explore PE in PS in Canada). Note that McDowell and Newell (1987) suggested this method as a way of confirming the content validity of a questionnaire when studying an unknown or new phenomenon in a large organizational or geographic context, such as in our case. McDowell and Newell (1987) suggest that questionnaire dimensions are better developed if they are defined and set up on the basis of the consensus, representativeness, relevance, and exhaustiveness of the constituent items of the concept or the topic to be studied [38, 39].
Based on their knowledge of and expertise in Canadian PE in PS, the expert advisory group assessed all the dimensions of the entire initial tool, item by item. When certain items were unclear, incomplete, or inconsistent, the expert advisory group proposed how they could be corrected or further developed. Through the assessment and adaptation process, the expert advisory group was able to create, modify, and adjust certain items to reinforce the initial tool’s (the translated English version) validity and consider the latest trends in PE in PS as practised by CHOs. For example, the initial translated English version tool did not have questions covering the leaders' own perceptions of change, the methods, and the techniques used by organizations to measure and assess the PE in PS outcomes and change. The initial translated English version tool review sometimes led to minor changes to items or simply adaptations, and at other times it led to the creation of new items based on new trends in PE in PS. For instance, a new item was created on the CHOs' managers' perspectives on the impacts of PE in PS and which factors most influence PE in PS, whether by enhancing or limiting PE in PS. Other new items were created in the section on demographic characteristics and information on the respondents' organizations, such as level of experience with the PE approach, competencies in the field, and level of PE understanding.
Findings: The assessment and review of the initial translated English version tool was the key stage in the entire tool development process. This resulted in the creation of a complete adapted version of the tool consisting of 75 items in 5 sections on the following 10 dimensions: (1) demographic characteristics, (2) experience level, (3) incentives, (4) strategies, (5) level of intervention, (6) structure and resources, (7) activities, (8) factors, (9) impacts, and (10) improvements (see Figure 3).
Step 4.2: Assessment and adaptation of the wording of the tool for pan-Canadian use:
Methods: In this step, all the members of the expert advisory group carefully examined the wording of every aspect of the tool and suggested improvements when they found inconsistent expressions and vague vocabulary. In addition, two members of Accreditation Canada (AC) performed a careful reading and a deep examination of the tool’s wording, based on their knowledge of and expertise in Canadian healthcare evaluation standards. Here again, a consensus-building approach was adopted to integrate the feedback and the suggestions provided by all the expert advisory group members on the tool's wording and language.
Findings: At the end of this step, our tool was formally defined as pan-Canadian. This step allowed us to adjust its wording to ensure that it would be understood by all CHOs. The expert advisory group’s comments on the adjustments made to the tool’s wording are found in Appendix A. Please refer to Figure 3 and Table 4 for more information on the changes made to obtain the final version of the pan-Canadian PE in PS survey.
Step 4.3: Editing and assessment of technical aspects
Methods:
This stage was focused on placing the questionnaire on the online platform, testing the technical aspects, and revising the questionnaire before the pilot test. The research team received specific training from the information technology staff at CPSI on how to manage and edit on the online platform. Then the online tool was internally tested by the members of the research team.
Findings: At the end of this stage, the questionnaire was set up online, approved by the working group, and considered ready for use in the pilot test. Please refer to Table 4 to see how the questionnaire was developed and adapted from the initial version (French version) to the final version (pan-Canadian version). In addition, please refer to the following link for the pan-Canadian survey, available on the CPSI online platform: https://survey.patientsafetyinstitute.ca/n/zz16p.aspx
Step 5: Pilot testing and pre-validation of the tool
Method: In this final phase, we tested and validated the tool among real CHOs. To this end we asked members of the National Health Engagement Network (NHEN), a community of practice, to participate in our pilot test. Once some members had agreed to participate and had given their consent to help test the survey, an email was sent to them explaining the instructions for completing the questionnaire along with the link to the online survey.
Findings: The questionnaire was completed in September 2020 by six organizations: 2 in British Columbia, 2 in Ontario, and 2 in Newfoundland. Among these 6 organizations, 3 have a mandate to provide acute care, 2 provide long term care, and 1 is specialized in mental health care. For each organization, the tool was completed simultaneously by a team of three members (one manager in charge of quality and risk management, one person in charge of PE, and one patient advisor). The average time to complete the questionnaire was 52 min, with a maximum of 67 minutes and a minimum of 27 minutes.
Once all the responses had been received, the research team exported and analyzed the results on CPSI’s online platform dedicated to the collection and processing of survey data. Based on the pilot test results and participants' comments, the research team made the necessary adjustments and amendments directly in the tool and emailed it along with the pilot test results to the members of the expert advisory group for final review and approval. Then, all expert advisory group members reviewed the pilot test results, revised the entire tool, approved the changes and adjustments, and gave their final approval. Following the pilot test, the tool was considered ready to be used officially in the final validation step: “at the pan-Canadian level.” Table 5 presents the tool items that were modified, adjusted, or adapted based on the pilot test results.
Table 4 shows the final sections (layout) of the pan-Canadian PE in PS tool.