Assessing the Quality of Deliberative Stakeholder Consultations with Pediatric Palliative Care Clinicians and Oncologists in Canada

Objective—This paper presents ndings from a quality assessment of deliberative stakeholder consultations with healthcare professionals in Canada on the implementation of a precision diagnostic for life-threatening pediatric brain tumors. Background—Advanced understanding of the basic biology and pharmacogenomics of pediatric brain tumors portend a clinical future in which oncologists base their clinical decisions in large part on results from laboratory derived tests (LDT) using next generation sequencing. Less is known, however, about how interprofessional healthcare teams perceive the opportunities and challenges of adopting LDTs in clinical practice, or how to best communicate LDT results to pediatric patients and their families. Deliberative stakeholder consultations present a promising alternative to traditional deliberative democratic methods. They allow researchers to unveil normative ethical and social values underpinning a forthcoming policy or new standard of care from the perspectives of key stakeholder groups, as well as make practical recommendations relevant for implementation. Methods—Using the DeVries framework for assessing the quality of deliberative process and information, we analyzed data from 44 post-consultation evaluation surveys from pediatric oncology and palliative care teams at two tertiary pediatric healthcare centers in Canada. Medians/means based on a 10-point Likert scale are reported. We also conducted turn-taking analysis and word-contribution analysis from the text transcriptions of each deliberation to assess equality of participation. LDT where median


Background
Tumor genotyping and pharmacogenomic testing have together enabled novel understanding of the basic biology and treatment responsiveness, respectively, of some pediatric cancer types (1). Brain tumors are the leading cause of cancer related death in children and malignant gliomas contribute an important component to this number (2). In a seminal 2017 paper, researchers characterized 85 different H3 K27 mutations associated with diffuse midline gliomas, an especially aggressive brain tumor in children. This discovery prompted development of a tumor-speci c laboratory derived test (LDT) with plans to implement the test as a standard diagnostic procedure and basis for development of treatment plans. H3 K27 mutations are resistant to all known therapies, and detection using the LDT carries a universally fatal diagnosis with no curative treatment currently available. Palliative care is thus currently standard therapy for the ~ 20% of glioma patients with a con rmed K27M variant.
Despite advances in understanding tumor biology and LDT development, sparse psychosocial research has investigated the ethical and social impacts of introducing an LDT for diagnosing K27 mutations in the clinic in standard practice, nor how to best support clinician communication with patients and their families in light of the catastrophic diagnoses this LDT could inform. The dearth of empirical evidence supporting patient-clinician communication and shared decision-making is generally identi ed (3) as a barrier to improved pediatric palliative care delivery. (4) We conducted deliberative stakeholder consultations with healthcare professionals across two large, pediatric academic medical centers in Canada to ll this aforementioned knowledge gap. Deliberative democratic methods rest on the idea that stakeholder communities, both lay and expert, are holders of experiential knowledge, and this knowledge is necessary for shared decision-making where new policies are in development (5,6). The deliberative stakeholder consultation used in this study is a hybrid citizen jury and structured focus group (7,8) in which deliberants freely debate the ethical, social and clinical implications of policies governing an emerging technology. Two outputs, deliberative and analytical (9), are systematically studied in the stakeholder consultation. The former are syntheses of collective decisions, opinions and statements from deliberants on the policy issue at hand, with limited attention to the discursive elements underpinning them. Analytical outputs, in contrast, are those which treats participants' statements as socio-cultural products, contingent on the discursive context within which they are produced" (9). Like all deliberative democratic methods, the consultations allow researchers to unveil normative values speci c to a forthcoming policy-or standard of care in medicine-from the perspectives of key stakeholder groups, as well as make practical recommendations for implementation (10)(11)(12).
Whereas myriad survey methods can be applied to assess whether and to what degree understanding and behavior can change (13), "if the deliberative process is to be trusted, we need to know more about what happens as people deliberate" (14). Without quality assessment of deliberative exercises, policy recommendations could disproportionately re ect the perspectives and interests of dominant actors; reinforce power imbalances and hierarchies; as well as mask values from stakeholders who may be more vulnerable, but no less essential to the policy development process (3). Studies which report quality assessment data alongside substantive ndings from the deliberative consultations demonstrated utility in four distinct quality metrics with at least Cronbach's alpha score of internal consistency reliability of > 0.7 (15,16).
To this end, we assessed the deliberative quality of six stakeholder consultations on the basis of process and information using criteria adapted from a validated survey developed from DeVries et al (15). We modi ed the authors' approach for assessing a third component of deliberative quality, reasoning, using ethnographic observations and aim to demonstrate in this article how deliberative quality is not compromised in the process. We describe elsewhere our rationale, ndings and the contributions this modi cation makes to the deliberative methods literature (17).

Methods
A post-consultation evaluation survey developed by DeVries et al to assess quality of information and process was administered to all deliberants between March 2015 and August 2016. Deliberants rated their deliberative experience and self-reported learning on a 10-point Likert scale. An expert facilitator gave an informational presentation at the start of the consultation, which covered background evidence on the current i) incidence, prevalence and burden of disease of pediatric high-grade glioma; ii) mortality data for speci c high-grade gliomas variants; and iii) sensitivity and speci city of a newly developed LDT to detect high grade glioma variants.
Deliberants were invited to ask the facilitator any clarifying questions in response to the information presented, followed by an open discussion of the following: 1. What information would you need to optimally communicate the results of the LDT so that the patients and their families understand the implications of treatment?
2. Would this change if the patient has the terminal mutation?
3. What concerns/problems do you think the families will have in accepting treatment plans based on the LDT?

What issues do you foresee for the implementation of this test?
The deliberative stakeholder consultations adopted in this study were modi ed from existing models in the literature (18) to accommodate for the extreme time constraints of healthcare professionals, and for the discreet power dynamics that can often manifest between more senior and junior clinicians during deliberative exercises (17).
Likewise, the hierarchal structures characteristic of subacute medical specialties like pediatric hematology/oncology and palliative care motivated us to conduct a separate assessment of participation equality, a subcategory of deliberative process (12) in the original De Vries framework. We rst transcribed all deliberations verbatim, and then conducted turn-taking analyses on the text transcriptions from each deliberation to achieve this.
We counted a turn anytime there was a change in speaker, and totaled the words spoken during their turn. We considered a deliberative session to have achieved proportionate participation when no one individual took more than 40% of total turns or spoke more than 40% of total words.

Analysis
Descriptive statistics, including group mean, median and range were calculated from deliberant ratings. We compared group means across all six deliberative stakeholder consultations to determine whether, and to what extent deliberants perceived the quality of the deliberative information and process in uenced their views of LDT implementation. We established two thresholds, more than 20% and more than 40%, to capture unequal participation in turn taking or words spoken based proportionately on the total N for each deliberative event.

Participants
Compared to existing models, the deliberative stakeholder consultations we conducted were practically focused around a speci c problematic and an appropriate course of action involving a purposively selected group of experts who would enact this change. Because deliberative stakeholder consultations aim to assess implementation in a narrow context (8), the clinical teams recruited to participate our study were unique to those 'mini-publics' that interact with glioma patients and their families, speci cally (19). We purposively recruited 44 clinical staff at two tertiary medical centers in Canada where nearly all regional, pediatric glioma patients are referred for specialty care, and where researchers planned to launch a pilot program of the LDT in the near future. We obtained survey data from just oncology teams (N = 18), just palliative care teams (N = 7) and from a mixed group of palliative care and oncology teams (N = 23) across both hospitals. Clinical teams comprised of physicians, nurses, social workers, medical students, and fellows.

Process
According to the DeVries et al framework, process refers to the degree of facilitation, equality of participation, participant engagement and feelings of respect as self-reported by the deliberants. Three survey questions probed deliberants' perspectives on the deliberative process (Table 1). Median ratings for feeling that one's opinions were respected by the group ranged from 9.5-10 across all deliberation groups. Ratings were similarly high in response to whether deliberants felt listened to and whether the process was fair, both ranging from 9-10. Relative to palliative care and mixed teams, oncology teams reported the lowest average and median ratings on each of the four process-related prompts. Oncology-only and mixed teams at Hospitals 1, and oncology-only teams at Hospital 2 were also less likely on average to abide by the group's nal position to implement the LDT (8.1, 8.35 and 8.7, respectively).

Equality of participation
We assessed a subcategory of deliberative process, equality of participation, by calculating the i) percent participation of deliberants across the six deliberations, tallied the ii) total number of turns taken and counted iii) total words spoken. Table 2 presents a summary of both the turn-taking and word counting analyses. Four of six deliberations achieved 100% deliberant participation, where participation entailed contributing to the discussion at least once. The greatest and lowest number of turns taken were analyzed during deliberations at Hospital 2 among deliberants in the mixed and oncology groups, respectively. Despite similar participation numbers, the palliative care-only group at Hospital 1 took nearly twice the number of the turns as its counterpart group in Hospital 2.
Although the palliative care only group at Hospital 2 reported the least number of turns, it registered the highest number of total words spoken of any group assessed in our study. Furthermore, this team was the only group where one deliberant contributed more than 40% of the total words recorded during the deliberation. The only other individual to take more 40% of total turns was a deliberant in the oncology only group at Hospital 1.

Information
Information refers to the use of on-site experts to present correct information, the extent deliberants take away new information, and understanding and applying this new information to inform one's opinion. Deliberants generally perceived the informational content presented during the presentation to be helpful, and that discussions with both the expert as well as with other discussants facilitated their understanding of the LDT. Palliative care teams reported higher median ratings for each of the rst three information-related prompts referring to the informational presentation, interaction with the expert and group discussion (from 9-10). Medians from oncology only teams were slightly lower on each of these items in comparison (8-10). With the exception of the palliative care-only team at Hospital 1, ratings in response to the question How much did attending the session change your understanding about the use of this new pharmacogenomics test in pediatric oncology? and How much did attending the session change your opinion about the use of this new pharmacogenomics test in pediatric oncology? were the lowest in the separated professional sessions at Hospital 2. Attending a mixed deliberative session, however, led to a measurable increase in these ratings-nearly 5 Likert points higher in some groups-when compared to attendance a session with oncology or palliative care teams alone. A similar result was also observed between the mixed group and the oncology only team at Hospital 1. The palliative care only team at Hospital 2 reported the lowest median change in understanding (2/10) and opinion (3/10) concerning LDT use of any surveyed group in the study.

Discussion
Deliberants generally agreed the quality of the deliberative process was both fair and respectful of their diverse opinions regarding the implementation of a new LDT test. The early stage of LDT research and development at the time we conducted the consultations meant that most clinicians were unlikely to have taken part in prior discussions about its implementation. Presenting this information in a comprehensive, yet accessible way was therefore critical to facilitating an informed discussion among deliberants and substantiates information as a necessary metric for deliberative quality (13). The modi cations made to the overall length of the deliberative session further underscore the importance of this deliberative quality assessment in speci c (11). Based on average ratings for the survey items that assessed information quality, attending the session had moderate effects on deliberants' overall understanding of the new LDT, and an even less effect opinion change on using the LDT in deliberants' clinical practice. Ratings among the palliative care only group at Hospital 2 demonstrated the starkest nding in this regard. Despite unanimous agreement that the presentation, expert question and answer session and peer discussion were adequately informative (medians from 9.5-10)-all survey items probing information qualitythe median rating of a deliberant's self-reported change in understanding and opinion of the LDT was 1.75 and 3, respectively. One possible explanation for why palliative care teams were unmoved by the information is that, ultimately, the responsibility of conveying treatment options to the child rests with the oncologist.
Our nding is consistent with a randomized control trial comparing four different deliberative methods in healthcare/health policy decision making. Brief citizen deliberations, most akin to the stakeholder consultation model adopted in this study, resulted in no statistically signi cant change in deliberants' personal opinions/preferences on healthcare decisions according to the study authors. They further note, "one may be more willing to accept the importance of knowing about medical evidence…before one would accept that evidence is more important than preferences. Deliberation had an impact on the "lower bar" of perceived importance but stopped short of the strongest changes in attitude in this domain" (16).
Deliberations within the mixed group attributed higher information quality ratings from those reported in the palliative care-and oncology-only groups at Hospital 2 but lower at Hospital 1. Several hypotheses may help to explain this nding. It is well documented in the literature that palliative care teams are more often consulted as a child's condition deteriorates and where available. They are seldom included as part of the interdisciplinary oncology team at diagnosis or early in the patient's care journey (20)(21)(22) when research demonstrates their involvement can greatly improve pain/symptom management and health-related quality of life, among other possible clinical bene ts (23). It is possible that low ratings in response to whether the sessions changed professional understanding or opinion may be relative to how deliberants perceive the integration (or lack thereof) between palliative care and oncology teams at the diagnostic stage in which the LDT will be implemented. This nding corroborates what has been found in the patient decision-making and communication literature speci c to the care of critically ill children with cancer (24).
There is some debate in the literature on the extent to which unequal participation deters from the quality of deliberative outputs (25). Our equality of participation assessment demonstrated that turn-taking was not disciplinespeci c nor correlated with group size. We observed disproportionate participation-more than 40% of total turns taken and words spoken-in two of the six deliberative sessions ( Table 2). While it does not invalidate the perspectives or the deliberative outputs of these sessions (9), this nding is important to disclose when making recommendations that the perceived opportunities and barriers to LDT implementation were largely driven by perspectives of select individuals. The disproportionate participation, namely at Hospital 1, did not however adversely affect perceptions of procedural fairness and respect as evidenced by the comparably higher ratings on these items in the survey reported earlier. Democratic deliberation is a transformative research process intended to draw out opinion, identify and potentially resolve points of con ict, and inform action. For the practical purposes of initiating discussion and informing action, discussion need not necessarily be proportionally equivalent. Because the discussants were generally in agreement that a fair discussion took place, it is reasonable to believe no major lingering issues were left unaddressed, and deliberants might have reported them elsewhere in the survey.

Limitations
Our ndings should be considered in light of several limitations. First, we targeted recruitment of healthcare professionals at the few tertiary academic health centers where the newly developed LDT would be piloted. Albeit a small sample size, this recruitment approach achieved the speci city desired to identify implementation challenges among those on the frontlines of implementation in the nearest future. It is therefore also possible that our responses may re ect a social desirability bias as a result. The deliberative stakeholder consultation itself is vulnerable to several methodological critiques, namely that ndings can disproportionately re ect the viewpoints and perspectives of the most dominant voices. While an imperfect proxy, our quantitative measures of process and information using a validated survey instrument, and equality of participation by analyzing turn-taking and deliberative quality, helped to determine the nature and extent of any power imbalances in the consultations.
Qualitative interviews could, in addition to evaluation surveys, provide richer qualitative evidence for assessment, particularly deliberative process and equality of participation criteria. Lastly, deliberative forums on highly sensitive topics have been criticized for their potential to engender further polarization. Our decision to rst conduct separate, followed by mixed deliberations with oncology and palliative care teams attempted to reduce this polarization, but which may have contributed to underestimates in both information and process quality. Indeed, dominance was a concern for the two discipline-speci c deliberation groups at Hospital 1 when compared to either of the mixed groups.

Conclusion
Cumulative results from our assessments of deliberative quality demonstrated that deliberants agreed the i) overall quality of the deliberative process was fair, the ii) the opportunity to receive quality information from an expert about the implementation of a new LDT and to discuss this information with others was helpful, and that iii) participation was proportionate among deliberants in at least four of the six deliberations. We found that equality of participation was challenged in two deliberative settings, whereby one deliberant took more than 40% of turns, and in another instance one deliberant spoke more than 40% of total words. These imbalances did not, however, appear to negatively affect deliberants' perception of fairness or respect during the session according to survey results. A forthcoming publication quali es how, and to what extent disproportionality in participation in uenced deliberants' conclusions regarding the barriers and opportunities of LDT implementation. Assessing deliberative quality not only enhances the rigor and trustworthiness of substantive outputs from deliberative exercises, but it also enables empirical study of how differences in quality may in uence consensus building processes. More contributions to the methods literature are needed to investigate these differences systematically before making such assessment part of standard research practice.
Abbreviations LDT Laboratory derived test Declarations Ethics approval and consent to participate This study was approved by the McGill University Institutional Review Board. All participants provided informed, written consent prior to each deliberative consultation.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Consent for publication
Not applicable.

Competing interests
All authors declare that they have no competing interests.
Funding Support for this study was provided by Genome Canada, Genome Quebec and the Canadian Institutes of Health #236825. The funder had no involvement in the study design, data collection, analysis or manuscript preparation.
Authors' contributions GB conceptualized the study and recruited participants. VR, CL and GB collected, analyzed and interpreted data from the deliberative stakeholder consultations and collaborated in writing the manuscript. All authors read and approved the nal manuscript.  (10) 10 (10) 10 (10) Do you feel you were listened to by your facilitator?