Development of the Patient and Public Involvement Questionnaire (PPIQ) for Canadian Drug Funding Committees

Background There is limited evidence evaluating the impact of patient and public involvement in setting priorities for healthcare, specically drug funding recommendation committees. We describe the development phases of the Patient and Public Involvement Questionnaire (PPIQ). Methods The development of the PPIQ was informed by previous work, which established nine criteria to evaluate patient and public involvement. The PPIQ was developed with a multi-method, multi-phased approach: 1) item generation and renement (item bank creation, user feedback sessions); 2) sensibility testing (using Feinstein’s criteria and structured interviews); and 3) pilot testing with drug funding committee’s in Canada. Results In phase one of development, a bank of 846 items were derived from key informant interviews and a literature review. Guided by the nine evaluation criteria, an initial draft of the PPIQ was created using the item bank. Two user feedback sessions (n=7) resulted in further revisions of the PPIQ. In phase two, participants (n=21) completed a sensibility questionnaire with a median score 6 out of 7 on 80% of items. Interview participants (n=14) articulated the PPIQ was clear and appropriate. In phase three, response rate for pilot testing the PPIQ was 25% (n=14) and the average time for participants to complete the PPIQ was 00:19:00 minutes (SD ± 13:46). Conclusions The methods used for the development of the PPIQ including qualitative interviews, literature review, user feedback, sensibility and pilot testing have resulted in a questionnaire that may be used with committees making drug funding recommendations in Canada.


Background
There has been a growing global commitment by policy-makers and providers to improve the quality of health-related decision making by incorporating greater patient and public involvement (1)(2)(3)(4)(5). However, despite this increase in stakeholder involvement, there is limited evidence of how this involvement affects health care priority setting (6,7).
Evidence has shown that issues raised by patients and patient advocacy groups are re ected in drug-related health technology assessment recommendations and hospital-based assessments (8,9). This evidence demonstrates that patients and patient advocacy group perspectives were often considered and that they enhanced the content and framing of the recommendations (8,9). However, inclusion of patient insights is only one facet of measuring meaningful participation; it does not address for example, the nature and quality of their participation in the process of committee deliberation (8,9). Evaluating the effect of patient and public involvement poses several challenges. These include determining and evaluating "successful" involvement, accounting for the effect of power differentials between different types of experts, and determining whether resources to support involvement are su cient (10).
The aim of this paper is to describe the development of the Patient and Public Involvement Questionnaire (PPIQ), a tool to evaluate the degree of involvement of all committee members, including patients and members of the public, in decision making within the speci c context of Canadian committees that make formulary listing recommendations. Our phases included the following: 1) generating and reducing PPIQ items (i.e., questions); 2) assessing sensibility, and 2) pilot testing of the PPIQ.

Methods
We used both quantitative and qualitative methods to develop, re ne, assess, and pilot the PPIQ with stakeholders involved in Canadian formulary recommendation processes (Fig. 1). Consistent with the principles of the process, we included the perspectives of committee members, patient group representatives, public drug plan employees and academic experts (national/international) in patient and public involvement in healthcare.

Conceptual Approach
The development for the PPIQ builds upon previous work by authors ZRY and AMB who developed nine criteria to evaluate patient and the public involvement in health care resource allocation decision making (10). In this work, authors ZRY and AMB conducted a literature review of studies describing the development or evaluation of questionnaires to measure public or patient involvement in public decision making (including non-health related elds). Additionally, the authors conducted key informant interviews with representatives of patient groups, past or present government employees, representatives from Canadian provincial Ministries of Health, advisory committee members and industry personnel. These nine criteria, outlined in Table 1, guided the re nement and organization of PPIQ throughout development. The purposes of patient and public involvement should be clearly de ned. This criterion may de ne whether a member's role is to represent the broader public view, the patient view, or both. Furthermore, rationale may de ne whether the objective of involvement is to consult with external patient advocacy groups or to actively partner with them in making decisions.

Representation of Relevant Views
A full range of perspectives should be represented. This criterion may assess whether patient and public members are su ciently representative of their constituencies and whether patient advocacy group submissions that are received from several groups adequately represent diversity.

Fair Decision-Making Processes
The processes and procedures used by the committee should ensure fairness in decision making from the perspective of patient and public representatives.

Su cient Support
Patient and public representatives should have su cient support to ful ll their responsibilities.

Legitimacy of Committee Processes
Committees' processes should be accountable and transparent. This criterion may assess whether participants understand how decisions are made and whether stakeholders are able to appeal decisions Adequate Opportunity for Participation Committees should have processes that allow public and patient representatives the opportunity to participate fully in discussions and to address power differentials.

Degree of Participation
Committee deliberations should include meaningful patient and public participation. This criterion may assess the level of participation of public and patient members in decisions and the number and quality of patient advocacy group submissions.

Noticeable Effect on Decisions
Public and patient involvement should materially in uence committees' recommendations. This criterion may assess how many decisions would have led to different conclusions had patient or public involvement not been present

Considerations of E ciency
Committees should have su cient resources to ensure effective participation, which should be used e ciently. This criterion may assess the time and money needed to make recommendations and the value obtained relative to the resources expended. Yunger & Bayoumi, 2017) were used in the current work to inform the rst phase of development for the PPIQ. The interviews and literature review provided a dataset that we used to develop items. We coded the dataset in an inductive line-by-line manner. We considered each code individually to develop the preliminary list of items (i.e, the item bank). Two members of the research team (ZRY and LV) reviewed the item bank to reduce the number of redundant items, identify missing items, and ensure items adequately addressed the purposes of evaluating patient and public involvement based on the nine criteria of public and patient involvement.

2.
1.2 User Feedback (Focus Group). We conducted one in-person focus group discussion with Canadian Agency for Drugs and Technologies in Health (CADTH) staff, representatives from patient groups, members of drug advisory committees, government employees, and industry personnel to capture stakeholder feedback on the PPIQ (Version 1). We identi ed informants from websites listing drug advisory committee members, and suggestions by staff in Ministries of Health. Written informed consent was received from all participants either in person or electronically.
Focus group participants were asked to review the draft PPIQ (Version 1) eliminate redundant items, add missing items, and ensure each of the nine criteria of public and patient involvement was adequately captured in the questionnaire. Additionally, we asked participants to assess ambiguity and to check for double-barreled items. Con icting views on which items to retain or delete were addressed at this stage by retaining these items. The nal decision to retain or delete items was made after the focus group data was analyzed. The focus group discussion was audio-recorded and transcribed. Two team members (ZRY and LV) analyzed the focus group transcript independently using a qualitative thematic approach resulting in suggestions to add, retain or remove items from the PPIQ. Any discrepancies in coding were discussed and resolved through consensus. We revised items based on recommendations from the focus group to result in a re ned PPIQ (Version 2). Participants were also asked to provide electronic feedback by reviewing items within the PPIQ as a Microsoft Word document, which was emailed to each participant. Participants were asked to make comments and track changes throughout the PPIQ Word document. Two team members (ZRY and LV) coded the open-ended survey responses and discussed and resolved any discrepancies in coding through consensus. The comments and track changes by each participant helped inform further re nement of the PPIQ not captured in the survey responses. We revised the PPIQ based on the online surveys and electronic feedback to create PPIQ (Version 3).

User Feedback (Online Survey). A second group of participants
Phase 2: Sensibility Testing 2.1 Sensibility Questionnaire. A total of 115 emails were sent out, of which 99 were delivered to stakeholders (35 committee members, 24 patient groups representatives and 40 academic experts) requesting their participation in the sensibility testing and 21 participated. The average age of participants was 50.9 years (standard deviation ± 12.3), and the majority identi ed as male (n = 13, 62%). Six of the participants were members of drug reimbursement committees, of whom ve were professional members and one was a public member. Eight participants were patient group representatives, and four were academic experts in the eld of patient and public engagement. Three participants remained anonymous.
Twenty-one participants reviewed the PPIQ and completed the sensibility questionnaire. The ve analyzed categories, including purpose and framework, overt format, face validity, content validity, and ease of use, were re ected by 9 items in the sensibility questionnaire. Table 3 provides a summary of responses by all participants and also by the different categories of participants. The median scores on the sensibility questionnaire were ≥ 5 (out of 7) for all 9 items (100%) reaching our criteria for sensibility. Three participants were anonymous and did not provide their role on committees. Only one public member completed the sensibility testing. Overall, the item with the lowest score (5 out of 7) was the participants' perception of the time to complete the PPIQ (Table 4). Considers the selection of items including missing item, and inclusion of unsuitable items, as well as, appropriateness of response options.

Ease of use
Considers the ease of administration and use of the questionnaire from the users and researchers' perspectives. *One public member is included in the overall analysis. Three participants did not report their profession, which are included in the overall analysis. *One public member is included in the overall analysis. Three participants did not report their profession, which are included in the overall analysis.
2.2 Sensibility interviews. Of the 21 participants who completed the sensibility questionnaire, 14 agreed to a follow-up sensibility interview (three academic experts, four committee members from drug reimbursement committees in Canada, and seven patient group representatives). Interview data was coded according to Feinstein's ve criteria. During the analysis and coding an additional theme emerged: "clear terminology".
Overall purpose and framework of the PPIQ. Participants thought the PPIQ addressed the overall purpose, to evaluate patient and public involvement on committees was appropriate as one participant stated: "I felt that the survey was actually pretty well put together" (INT 7).While most participants thought the PPIQ addressed the overall purpose, many wanted the questionnaire to identify the purpose up front in order for the participant to understand the reason for completing the survey: "Knowing the purpose of why and then circling back and showing them the outcomes, I think those are the most important pieces." (INT 8).
Overt format. Participants highlighted the need to include a progress bar, which identi es how far along one is in the survey: "I nd that useful. It's just a bit of telling people that you're making progress" (INT 4). Another participant suggested that the PPIQ have a start stop option to aid in the ease of its use: "Some people just for various reasons might not be able to sit down and do it within 20 minutes. So I would absolutely have the option that you can stop and start" (INT 11).
Face validity. Most participants thought that the PPIQ was clear. As one participant said, "I think the questions are … [written] in very easy to understand language. And I think the way they're written, they would resonate with people" (INT 10). Another participant stated, "I don't recall any particularly confusing [items]" (INT 5). Participants reported that the content in the PPIQ was comprehensive: "I thought that it was really thorough. And I thought that it did a really good job asking about people's thoughts on the public and patient involvement process from many different perspectives" (INT 3).
While participants agreed the PPIQ was comprehensive, they noted that this resulted in a lengthy questionnaire.
Content validity. Participants did not identify any missing items however, a few identi ed some redundant items. There was discussion around "whether public and patient, consistently need to be pulled apart" (INT 3) across the items. Another participant thought "the questions about the chair of the committee ensuring that these perspectives are considered. It's a little bit redundant … do we really care that it's the chair that makes it happen or it's just part of the process?" (INT 5). Finally, one participant "wondered if maybe it [some items] could be consolidated" (INT 8). Participants also discussed reducing the number of response options.
Ease of use. Participants indicated that overall the PPIQ was easy to use. As one participant noted, "I think it owed well. Like it didn't feel like it was leading me anywhere. Which is important, right? It was clear and logical" (INT 8). A concern that they raised was the literacy level of the questionnaire: "I mean I think that anybody with below a high school education would have trouble with this questionnaire" (INT 4). However, another participant said "the language is also re ective of or applicable to who might be lling it out" (INT 11). One participant highlighted that each question was needed: "I understand you'd like to … shorten it a little bit. But truthfully, there's probably not a lot I would eliminate. You know, even just scanning over it again, I mean I think they all ask different things" (INT 2).
Clear terminology. Some participants thought that further clari cation of terms and de nitions used within the survey was required. Participants highlighted the importance of providing a clear de nition for the term, "industry". For example, "And the other one too just in terms of language, you referenced industry in this as well. I mean I interpret industry … as … pharma. That needs to be spelled out or just de ned maybe at the outset" (INT 11). Another participant said that "I remember feeling that sometimes it was di cult to sort of differentiate … [between] patient versus public" (INT 5).
Based on the results of the sensibility testing combined with research team feedback, we revised the PPIQ (Version 4) prior to pilot testing. PPIQ (Version 4) was informed by the Phase 2 sensibility testing, prompting us to improve the face validity and the purpose and context of the PPIQ through more clearly de ning its purpose in the introduction of the questionnaire. Also, we more clearly articulated the distinction between public and patient stakeholders throughout the questionnaire. To improve ease of use, we reduced the response options on each question from seven to ve options, and because the PPIQ is designed to be administered electronically, we included a status bar to illustrate progress to improve user experience and the perceived length of the PPIQ.

Phase 3: Pilot Testing
A total of 55 committee members across Canada were contacted by email to complete the PPIQ (Version 4) of which 15 (27%) opened the PPIQ, and 14 (25%) participants completed the PPIQ. The sample consisted of professional drug committee members: physician (n = 7), pharmacist (n = 4) and academic/researcher (n = 3). The average age of participants was 52.4 years (SD ± 12.3 years), with eight (57%) participants identifying as male. The average time for participants to complete the PPIQ was 19:00 minutes (SD ± 13:46). Table 5 summarizes the pilot data within each of the nine earlier established evaluation criteria (Rosenberg-Yunger & Bayoumi, 2014). For seven of the nine evaluation criteria, the majority of participants (> 50%) agreed (inclusive of 'agree' and 'strongly agree') that their respective committee satis ed the criteria based on the items in the PPIQ (Table 5). Approximately 10% of all participants' responses were 'not applicable' for PPIQ items. Three items were identi ed as having over one third of the total sample answering 'not applicable': Question 18, "Patient member(s) adequately represent patient perspectives during our committee deliberation" (71%, n = 10, 'not applicable'); Question 19, "Public member(s) adequately represent public perspectives during committee deliberation" (43%, n = 6, 'not applicable'), and Q53, "Our patient group submission guidelines are easy to nd on our website" (43%, n = 6, 'not applicable').

Results
Phase 1: Item generation and reduction 1.1 Item Bank. The preliminary item bank had a total of 846 items. This was re ned internally resulting in an initial draft of the PPIQ that had 85 items. After reviewing the 85 items with the larger team, we reduced the PPIQ to 71 items, which was then organized according to the nine pre-speci ed criteria (Table 1) and presented to stakeholders for feedback.
We then organized the item bank putting similar items together (i.e., patient and public involvement, patient group submissions, role of the chair, etc.). We created brief instructions which outlined the purpose of the PPIQ, and a Likert scale with seven response options labelled from very strongly agree (0) to very strongly disagree (7). This resulted in an initial draft of the PPIQ (Version 1) to be presented to users for feedback.  (Table 3) were added. Think back to a time when patient input was not necessary in the drug recommendation process, why was this so?

Question 4
Can you think of a time when public and patient input was necessary in the drug recommendation process but did not add any value to the deliberation? Why was this so?
1.3 User Feedback (Online Survey). Given the low attendance during the rst stakeholder feedback session via the focus group, we conducted the online user feedback data collection to gain additional feedback on the revised PPIQ (Version 2). A group of 14 individuals were contacted for the second stakeholder feedback session. Four participants (a committee member, a patient group representative, an industry employee, and a knowledge user partner representative) responded to the online survey. Participants identi ed missing questions including items around compensation and "adequacy of training on HTA [health technology assessment], including the clinical and economic reviews" (P2). Results from the online survey and electronic feedback were reviewed by the research team and knowledge user (CADTH) resulting in a revised PPIQ (Version 3).

Discussion
In this study, we used a multi-methods multi-phased approach to develop and re ne a questionnaire to evaluate patient and public involvement in committees making recommendations regarding public funding of drugs. The purpose of the PPIQ is to evaluate the level of 'successful' patient and public involvement based on nine criteria allowing for a dynamic evaluation of public and patient involvement across different drug recommendation committee contexts (10). By eliciting stakeholder feedback, we were able to ensure that the PPIQ had face and content validity and that it met the expectations of potential users, including patient group representatives, industry, and patient and public committee members. Based on participants' responses to the sensibility questionnaire and on interview data, the content was appropriate, and the items were necessary and contributed to adequately evaluating public and patient involvement on drug reimbursement committees. Our pilot indicates the feasibility of delivering the PPIQ electronically (based on response rate and time to complete), and demonstrates the applicability of questions across multiple committees. While the sample of the pilot was small, the method for summarizing items within criteria may be useful for future implementations, such as testing reliability or construct validity of the PPIQ with drug funding committees in Canada.
The sensibility interview results indicate that the practicality of implementing the PPIQ among different committees may vary based on the number and type of members, as well as available time and resources and competing committee priorities. Overall participants involved in the sensibility testing reported that the PPIQ was easy to use; however, this criterion had the lowest median score, speci cally related to its length (median score of 5/7). The sensibility interview data suggested the length may be a limitation when implementing the PPIQ in the 'real world'. These pilot data suggested that on average it took participants approximately 20 minutes to complete the PPIQ, which may or may not be appropriate for committee members with competing priorities. Nonetheless, participants acknowledged the complexity of patient and public involvement in drug reimbursement committees, and in turn understood the need for the PPIQ to be robust and comprehensive.
The PPIQ is one of few questionnaires to measure public and patient involvement within formulary recommendations for public funding. Another questionnaire that may be used to evaluate public and patient involvement in our context is the Public and Patient Engagement Evaluation questionnaire (PPEET) (18). The PPEET was initially developed in 2011 to measure engagement broadly, allowing the questionnaire to be used in a variety of contexts in Canada with the objective of improving the quality of public and patient engagement. In contrast, the PPIQ focuses speci cally on drug reimbursement committees, and as a result provides an evaluation of member involvement that is speci c to the context drug committees in Canada.
Some limitations are present in our approach. Throughout our user testing (phase 1.2, 1.3) we had low participant numbers; however, this phase was also informed by the key informant interviews and literature review conducted previously by AMB and ZRY. In our sensibility testing, we had only one public member and no patient committee members, but we sought to generate patient and public perspectives through representation from multiple patient group members. When completing the pilot test of the PPIQ there was only representation from professional committee members (i.e., physicians, pharmacists, researchers, etc.), which highlights the need for further testing of the PPIQ with public and patient committee members.
Despite these limitations, the PPIQ was developed across multiple phases, with opportunity for a breadth of perspectives from across stakeholder groups. 9. Having completed the questionnaire do you think that it would be acceptable to others from the standpoint of understanding the questions, the time to complete and their acceptance of its comprehensiveness? 1234567 Unacceptable Poor Good Excellent Box 2. Sensibility Testing Interview Guide