The GASTROS study (www.gastrosstudy.org) is the first to bring together healthcare professionals and patients with the purpose of identifying outcomes to include in a COS for surgical trials in gastric cancer. The multi-language survey recruited a broad spectrum of stakeholders with different personal and professional experiences from over 50 countries across 6 continents. We aimed to examine whether certain stakeholder characteristics influenced how outcomes were prioritised and whether there were regional influences also. Our analysis from nearly 1000 survey participants suggested that little variation within the stakeholder groups exists. Similarly, when all stakeholders were categorised according to region or country income, significant differences were not identified. These are important findings which should serve to reassure researchers and patients that the resulting COS has sought and considered international opinion. Furthermore, these findings suggest that priorities within stakeholder groups and across regions are more aligned than may have been previously thought.
4.1 Planning recruitment to Delphi surveys
Few studies have previously examined factors which influence how stakeholders prioritise outcomes in the field of COS development. The BRAVO study explored this in the field of breast cancer reconstruction and found that priorities varied within patient and healthcare professional groups6. This led them to recommend careful participant selection for Delphi surveys by COS developers. These same differences, however, were not identified in our study. The BRAVO study’s healthcare professional stakeholder group was more heterogenous than the groups in this study (breast surgeons, plastic surgeons, nurses and psychologists grouped together) and so these differences may be expected. Furthermore, reconstructive breast surgery is a complex area which covers many different types of procedures. This may also account for the significant variation in outcome prioritisation by patients which was not mirrored in the GASTROS study. Similarly, a COS study in the field of bariatric surgery identified significant variation in outcome prioritisation amongst healthcare professionals13. Again, healthcare professionals in this study were heterogenous, which supports our strategy to separate surgeons and nurses into different stakeholder groups.
Achieving the ‘correct balance’ of representative stakeholders is an important consideration during the design phase. For example, knowledge of the patient demographic and which types of interventions are prevalent within that group, will enable researchers to recruit an appropriate number of stakeholders with those characteristics. With respect to the GASTROS study, the importance of seeking international agreement on core outcomes was identified at the conception stage and subsequently influenced the design of the prioritisation exercise. Our strategy for addressing the significant challenges associated with international involvement included 1) an international working group with regional collaborators, 2) translating surveys and 3) seeking the support of relevant patient and professional groups. Transparent reporting of methodological approaches adopted during COS development are of paramount importance. Ultimately, a COS will only achieve its stated goals if researchers use it. And whilst there are likely several factors which influence the uptake of COS, ensuring researchers have the confidence that the COS is relevant to them and has been developed through a methodologically robust process are likely to be important factors which influence uptake and dissemination14.
There are challenges in deciding how to sample participants for a Delphi study. Epidemiological studies, registries and audits provide descriptive regional or national information15–17. However, in the case of gastric cancer, these resources are not always complete or available. Consequently, the study team widened the promotion and enrolment into the Delphi to capture as many patients as possible. In our study, we demonstrated that there was not significant variation in outcome prioritisation within stakeholder sub-groups with respect to the characteristics that we examined. Consequently, whilst over 1000 participants were enrolled, it may not have been necessary to recruit such large numbers. This will likely guide our recruitment strategy during future planned stages of work when reviewing the COS and identifying outcome measurement instruments. Our experience may also help guide other COS developers as they consider the number of participants to recruit to their Delphi surveys. However, given some of our findings differed from those in the field of breast surgery reconstruction and bariatric surgery, more work is needed before broad recommendations can be made.
4.2 Variations within stakeholder groups
When regional variations across the three stakeholder groups were compared, the greatest differences in prioritisation were observed amongst nurses. For example, in four outcomes (pain, ability to undertake physical exercise, impact on mental health, need for additional intervention) different subgroups of nurses categorised them as ‘consensus in’ and ‘consensus out’. These outcomes seemed less important in LMIC and HIC settings within the nurse group. Understanding the reason for this is likely to be complex. It may be argued that this is simply because nurses are reflecting the importance that patients from these cultures or regions place on these outcomes as similar trends were seen amongst patients. Limited resource in LMIC settings which may affect follow-up may also play a role in understanding how important longer-term problems are in these regions. Further exploration using qualitative research methods may help understand these differences further.
In examining the differences between patient sub-groups, one would expect to see some differences given the number of characteristics that were examined. Despite this, only two outcomes (urinary complications and conversion to open surgery) were simultaneously categorised as ‘consensus in’ and ‘consensus out’ by different sub-groups. This finding suggests that despite the many possible influences on patient experience following gastric cancer surgery, there is not a significant variation in how health related outcomes are prioritised in this group. Surgeons had the greatest concordance with respect to outcome prioritisation. Overall, the observed differences in outcome prioritisation were small within each stakeholder group reassuring researchers using the COS that it is based on the views of a representative cohort of patients and healthcare professionals.
4.3 Impact of regional variations on outcomes automatically included in COS
The aim of a COS is to identify outcomes which are critically important across all stakeholder groups participating in the process. In the case of the GASTROS study, an outcome would only be automatically included in the COS if patients, surgeons, and nurses each categorise it ‘consensus in’. Ultimately, it is not possible to confidently assess how regional differences may have affected the final categorisation of outcomes which informed the consensus meeting. Participants in round 2 were shown the scores of all stakeholder groups from round 1 before being asked to change their score if they wish. To assess regional differences, Western participants, for example, in round 2 would have needed to see only Western stakeholder group scores from round 1. Furthermore, there are a number of other confounding factors which influence why participants change scores between rounds (see below) further making an analysis of regional impacts challenging.
Despite this, some assessments could be made. No outcomes categorised for automatic inclusion by all three stakeholder groups were categorised for automatic exclusion by a regional sub-group. And no outcomes categorised for automatic exclusion from the COS by all three stakeholder groups were categorised for automatic inclusion by a regional sub-group. This suggests that the regional differences in approach to management or patient outcome may not significantly influence how stakeholders prioritise outcomes
There were two outcomes (gastrointestinal functional problems and ability to complete treatment pathway) which were categorised for automatic inclusion by stakeholders from the West and HIC that were not included in the final list presented to the consensus meeting. Furthermore, some outcomes (surgery-related death, nutritional outcomes, bleeding, overall quality of life, anaesthetic complications) did not reach consensus for automatic inclusion by regional sub-groups yet were automatically included when the overall views of stakeholders were considered. This may bring some to the conclusion that different COS should be developed for different regions as some researchers may be collecting outcomes that were not deemed critically important in their region. However, researchers should be cognisant of the fact that their trials are internationally relevant and vitally important to the larger picture where evidence synthesis is concerned. From a different perspective, some researchers may feel aggrieved if outcomes which are critically important in their region are not eventually included in the COS. It is important to emphasise that COS are minimum reporting guidelines and that researchers are encouraged to report additional outcomes that they believe are important.
4.4 Strengths and Limitations
Strengths of this study include that it is novel and that is was able to recruit a large number of participants from many countries. However, there are some limitations which should be acknowledged. This was an analysis which was not powered to make definitive conclusions about relationships between sub-groups and how outcomes were rated. Therefore, the results should be viewed in this context. Furthermore, the sub-groups examined in this paper were chosen by members of the study team based on their extensive experience in the field of gastric cancer and their understanding of factors which may impact on stakeholder experience, perceptions and subsequently how outcomes may be prioritised. It is possible that other unexplored characteristics impact on how stakeholders prioritise outcomes. In addition, this study did not explore how different characteristics interact with one another to impact on outcome prioritisation (e.g. years since surgery and type of gastrectomy). Doing so would create results which would remove the focus from regional differences and would be difficult to interpret. Furthermore, there were significantly fewer patients from Eastern countries enrolled compared to their Western counterparts. This may have influenced how outcomes were categorised ahead of the consensus meeting. However, due to the interplay of other factors described above, reaching a definite conclusion about the degree of this possible limitation is difficult. This is an area that may benefit from further exploration using qualitative research methods.
Delphi surveys are an established method of reaching consensus in the design of COS1. They give participants the opportunity to reflect on their ratings from previous rounds before giving a final score. Only after this opportunity should all scores be analysed, and outcomes categorised ahead of the consensus meeting. During the process of rating outcomes in round 2 of the survey, participants are shown the results from each separate stakeholder group in round 1. The topic of why participants change their scores between rounds is an interesting one which has been examined elsewhere18. Through our previous analysis we identified that the reasons for changing scores provided by stakeholders were varied, including having the time to reflect on the question being asked, changing their minds on the importance, impact or usefulness of the outcome in question, and changes in personal experience of the outcome. In fact, the influence of other stakeholder ratings as a reason for significantly changing a score in round 2 was cited by only a minority of healthcare professionals and patients.
Another factor which may influence scores between rounds is attrition. Our strategy to keep this as low as possible, alongside other approaches to facilitate international participation in Delphi surveys for COS is a topic which will be described separately. Whilst overall attrition was 30%, the group this affected the most were nurses with nearly 45% attrition. However, the characteristics of those completing both rounds were not significantly different to those only completing round 1. Likewise, a statistically significant difference was identified in the characteristics of surgeons completing both rounds who were predominantly Western and from HIC compared to the balance of surgeons completing round 1. It could be argued therefore that retaining a greater number of Eastern and LMIC surgeons may have led to slightly different survey results. However, whilst statistically significant, this difference is unlikely to be clinically significant given that the number of surgeons not participating in round 2 was relatively small.
The GASTROS Delphi survey recruited a broad spectrum of international stakeholders to produce a list of outcomes which should be included or excluded from a COS and others which required further discussion at a consensus meeting. Whilst some regional differences were highlighted, there was little variation within stakeholder groups and between regions with respect to how outcomes were prioritised. This may reassure COS users that the adopted methodology was robust and that the views captured during its development were representative. COS developers should carefully consider the characteristics of Delphi survey participants when planning their recruitment strategy.