For the multicriteria study of stakeholder preferences, five working groups (WGs) were created consecutively, each with specific objectives. All of them were face-to-face, except for WG4, which was via the Internet.
WG1 defined the general objectives, identified the groups of "stakeholders" or relevant actors that provide the preferences, and created a draft of criteria and sub-criteria. WG2 evaluated and agreed on the criteria and sub-criteria. WG3 was comprised of three subgroups: WG3-A, WG3-B, and WG3-C. Each of these groups independently, in parallel, and face-to-face weighted the criteria and sub-criteria according to their preferences using two different multicriteria methodologies: weighted sum (WS) and analytic hierarchy process (AHP). Two weeks after this weighting, a survey was sent by email to each individual regarding their preference for the results of the WS or AHP method. Via the Internet, WG4 weighted the criteria and sub-criteria only by the method with the highest preference in the survey. In this way, a new weighting of the criteria and sub-criteria was established through similar methodology, but with a larger sample in order to validate the results of the face-to-face WG3 (obtained with a small sample size) and guaranteeing significant conclusions. Finally, WG5 consisted of two independent academics, specialists in multicriteria, who analysed the results (Figure).
Working Group 1
WG1 consisted of six researchers: four were nephrologists and two multicriteria analysts. The group defined the general objective of the study and the structure and composition of the remaining groups. The general objective was to determine the relevant criteria and sub-criteria of haemodialysis treatment and their weighting according to the preferences of the stakeholders. The preferences of the stakeholders allow a "performance matrix" summarizing the measured preferences for each relevant criteria to be established and determine an "aggregation function" that allows weights to be combined consistently with stakeholders’ preferences. This function enables analysis of the results of the centres considered in the study, and establishes their individual qualification in an orderly and justified manner.
WG1 defined the requirements and number of actors that comprised WG2, WG3, and WG4, which included patients, clinicians, and managers. All participants were recruited on a voluntary basis. The patients should have been in haemodialysis at least three years and have exercised coordination tasks in some organization of kidney patients. They were recruited from such organizations, mainly ALCER (Asociación para la Lucha Contra las Enfermedades Renales). The clinicians had to be of recognized prestige and extensive experience, one of them a nephrologist, an internist, and a nurse. For the managers, three profiles were defined that should be present in each group: economic direction, medical direction, and health services researcher. Clinicians and managers were recruited mainly from the centres involved in the study, they were contacted via phone-call, e-mail or personal approach. WG2 and WG3 (WG3-A, WG3-B, and WG3-C) were each comprised of nine interested individuals: three patients, three clinicians, and three managers (total 36 individuals: 4 groups x 9 interested in each). WG3-A was located in Alicante, WG3-B in Segovia, and WG3-C in Zaragoza. WG4 was comprised of at least 15 stakeholders from each category (patients, clinicians, and managers) who were located in different parts of Spain and participated online.
The criteria and sub-criteria were identified sequentially in two steps. First, WG1 agreed on the draft criteria, and then WG2 agreed on the criteria. The criteria are the relevant factors for the evaluation and ordering of the different options (haemodialysis centres). These must meet certain requirements in relation to the MCDA methodology used (completeness, non-redundancy, non-overlap, and preference independence). The following principles were also considered for the selection of draft criteria: feasibility of its implementation, potential modifiability of the indicator, and impact on the patient.
WG1 defined the search strategy in PubMed/Medline, EMBASE, and Cochrane Library. The terms included were: haemodialysis, outcomes, registry, patient reported outcomes (and equivalents), and clinical guideline. Priority was given to the PRISMA clinical guidelines (Preferred Reporting Items for Systematic Reviews and Meta-analyses). Two WG1 members independently reviewed the literature results and proposed a first draft of the criteria and sub-criteria to the rest of the group. After a discussion in the whole group, the draft criteria and sub-criteria were approved. An "evidence-based" criterion composed of various sub-criteria was established. To determine this, the group decided to consider only the recommendations of level 1 in international clinical guidelines to provide the study with transparency and reproducibility. This decision was made in a manner consistent with the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) considerations. We focus on three clinical guidelines that provide an appropriate framework for the study: Kidney Disease Improving Global Outcomes (KDIGO, https://kdigo.org), European Renal Best Practice (ERBP, http://www.european-renal-best-practice.org) and Kidney Disease Outcomes Quality Initiative (KDOQI, https://www.kidney.org). Finally, the ultimate decision on the inclusion of each criterion was made separately by the majority of the group (at least four members of WG1).
Working Group 2
WG2 consisted of stakeholders (3 patients, 3 clinicians, and 3 managers) working face-to-face. The group used WS methodology to reflect the criteria and sub-criteria by carrying out a qualitative structured analysis of the draft criteria and sub-criteria prepared by WG1. The deliberation was recorded and two independent analysts from the group with four pre-established criteria (internal, external validity, reliability, and objectivity) contributed to validating the selected criteria.
Working Group 3
The face-to-face WG3 weighted the criteria and sub-criteria agreed upon by WG2. WG3 consisted of three subgroups, independent and parallel in time (WG3-A, WG3-B, WG3-C). Following a multicriteria approach, the following were performed sequentially: baseline weighting using the WS methodology, a structured debate, and a second weighting using two different multicriteria methodologies: WS and AHP. The purpose of the weighting was to elicit the preferences of the stakeholders for each of the criteria and the reasons for their preferences.
The WS is an additive model in which the stakeholder is invited to distribute 100 points proportionally to his preferences among the set of criteria and sub-criteria (total sum 100). For example, if stakeholder Patient 1 has to rank a group of four criteria according to his preferences, distributing 100 points among them, the weight could be 10 points for criteria 1, 60 for criteria 2, 10 for criteria 3, and 20 for criteria 4. The stakeholder establishes the weights for each criterion simultaneously. The AHP is a multicriteria technique in which, for each node of the hierarchy considered, the stakeholder compares in pairs the relative importance of the elements (criteria, sub-criteria, or alternatives) that hang from it according to the fundamental scale of Saaty [19]. The result in each node is a positive reciprocal square matrix from which the local priorities are obtained and a measure of the decision-maker's inconsistency when issuing their judgments. The Super Decisions program was used for this (https://www.superdecisions.com, sponsored by Creative Decisions Foundation). The results obtained are transferred in the same way to a distribution of 100 points to their preferences among all of the criteria and sub-criteria (total sum 100 points). For example, in the AHP model, if stakeholder Patient 1 weighs a group of four criteria using a pairwise consecutive comparison (criterion 1 with 2, criterion 1 with 3, criterion 1 with 4, criterion 2 with 3, criterion 2 with 4, and criterion 3 with 4), the comparison is made on a quantitative scale that reflects the importance of one criterion in relation to the other. Both criteria can be ranked equally. The Super Decisions program allows the weight of each criterion to be found, in which the sum of all criteria is 100 (e.g., 10 points to criterion 1, 60 to criterion 2, 10 to criterion 3, and 20 to criterion 4). In this way, the results obtained by the WS and AHP methods are comparable. The assessment of criteria requires a personal and collective reflective process based on the individual weighting and a collective structured debate of the stakeholders, reasoning their different interests and perspectives to outline the trade-offs between criteria.
Two weeks after the meeting of each WG3, a survey was carried out with each participant. They were asked which results (WS vs. AHP) better reflected their preferences (or none of them or both equally). The survey was conducted blindly via email (the interviewee answered ignoring the methodology, WS or AHP). Thus, the researchers determined which method best expressed the preferences of each stakeholder according to their criteria.
Working Group 4
WG4 again weighted the criteria and sub-criteria, but only by the method (WS) that best expressed the preferences of WG3 in the survey. This new weighting was performed to check the consistency of WG3’s results. WG4 reproduced the multicriteria appraisal process of WG3 via the Internet. It was composed of a minimum of 15 patients, 15 clinicians, and 15 managers. Thus, the method sequentially included a first baseline weighting (WS), structured deliberation, and a second weighting exclusively using the methodology preferred by WG3 (WS or AHP). An ad hoc website was designed in HTML5 with CSS, JavaScript and AJAX on the client side, and PHP with MySQL on the server side, tools that met the necessary requirements imposed by the methodology. The discussion via Internet was anonymous, but the category of the stakeholder was shown (patient, clinician, or manager).
Working Group 5
Finally, WG5 integrated two independent academic experts in MCDA, who analysed the results. The statistical study was carried out using SPSS software and consisted of the following phases: (i) analysis of face-to-face results by stakeholder category and by methodology (WS vs. AHP); and (ii) analysis of significant differences between face-to-face and Internet results by stakeholder category, a T-test of means and ANOVA methods have been used for statistical analysis and significance. A Bonferroni test have been used to prevent data from incorrectly appearing to be statistically significant, if necessary.