Establishing Consensus on a Screening Tool for the Neglected Sexual SIde Effects after Prostate Cancer Treatment: A Modified E-Delphi Study.

Abstract

Background

Neglected sexual side effects are a group of less common sexual side effects after Prostate that may present after Prostate Cancer treatment. There is currently no valid and reliable tool to identify these side effects. A modified Delphi study is an effective way of developing and validating such a screening tool.

Methods

A modified Delphi study was used to obtain consensus from a multi-disciplinary group of experts over three rounds. Ten statements were presented containing 8 close ended statements on individual NSSEs, and 2 open ended  statements on Psychosocial impact related to NSSE. Consensus was defined as a 75% strongly agree achievement on each statement, or the final statement evolution at the end of 3 rounds. Statement support in each round was determined by mean, standard deviation and range, after a numerical value was allocated to each statement during specific rounds. All three rounds were structured and suggestions and additions were incorporated in the statement evolution of the three rounds.

Results

Thirty five participants were invited, and 27 completed Round 1(RD 1), 23 participants completed RD2, and 20 participants completed RD3. All 3 rounds were completed in 12 weeks. Statement 1(sexual arousal incontinence), statement 2(climacturia) and statement 3 (orgasm intensity) reached consensus after RD2, and statement 9 (sexual dysfunction impact) and statement 10 (experiences) were removed after RD3. Statement 4 (orgasmic pain), statement 5 (anejaculation), statement 6(sensory disturbances), statement 7 (penile length shortening) and statement 8 (penile curvature) were finalised after the conclusion of RD3. Statement 1-3 were the most stable statements with the most support and least amount of disagreement. Statements 4-8 were less stable, but support for them improved over the 3 rounds. Statement 9-10 both had good stability, but the support indicated that they needed to be removed from the set of statements. Statement 5 had the poorest range due to an outlier opinion.

Conclusions

Consensus was reached on the statements making up the NSSE screening tool. Health care practitioners will be able to use this tool to identify NSSE after prostate cancer treatment. Further reliability testing on this tool is needed.

Background

Disability amongst men is high following their diagnosis and treatment for prostate cancer (PCa) (1, 2). The reported incidence of PCa globally was 1,3 million cases in 2017, but more importantly, this was responsible for 7,1 million disability adjusted life years in these diagnosed men (3). The average age of PCa diagnosis in South Africa is 68 years, and the average age of death due to PCa is 74 years (4). The risk of developing PCa increases exponentially after the age of 50 years for South African men, and older age and ethnicity (African Black men) are the most notable non-modifiable risk factors leading to more aggressive PCa (5). Treatment of localized PCa may include surgical (prostatectomy) and non-surgical intervention (radiation therapy)(6). These interventions may cause disabling side effects that include pain, incontinence and sexual dysfunction (2, 7–9). Only a fifth of men will reportedly ever discuss issues of sexual dysfunction with their health care practitioner after prostate cancer (10) and while they may recover from pain and incontinence; they will suffer debilitating and long-lasting effects because their sexual dysfunction remained undetected (1, 11).

The less common symptoms of sexual dysfunction after PCa treatment may present in the form of a variety of complications that are collectively referred to as “Neglected Sexual Side Effects (NSSE)” (12, 13). These NSSE drastically impacts the quality of life in many men (9). The complications range from anejaculation; change in penile length and curvature; urinary incontinence during sexual activity (climacturia); orgasmic disturbances that encompass anorgasmia; changes in orgasmic sensation; and painful orgasm, among others (12, 14). Sexual function is the quality indicator most strongly associated with outcome satisfaction after prostate cancer treatment (9) and sexual dysfunction is a predictor of bother and depression after PCa treatment (15, 16). This association remains evident for a minimum period of four years after the diagnosis of prostate cancer, even after correction for possible confounders (17).

A literature review indicated that there is only a few original publications on the NSSES after PCa treatment, but only two publications on how to assess them (12, 14). Both studies used an informal non-validated outcome measure to determine the extent of the NSSE. Other common PCa related sexual dysfunction outcome measures includes the Expanded Prostate Cancer Index Composite (EPIC) (18) and the International Index of Erectile Function (IIEF) (19). The EPIC and IIEF are both validated instruments and both were recommended at the Fourth International Consultation for Sexual Medicine in 2015 (11), but both these instruments only address general sexual dysfunction and there are no questionnaires to assist in the diagnosis of the NSSE after PCa. There was thus a need to develop and validate an instrument that will effectively and quickly pick up the NSSE after PCa treatment. The Delphi technique has proven to be a reliable measurement instrument to develop and to refine a new concept, and to direct future research (20). The Delphi technique is also a cost effective and efficient method to collect information from an expert panel of participants, and is ideally suited for electronic administration (21).

The aim of this study was to develop an instrument to be used as a self-administered clinical screening tool to identify NSSE after PCa treatment. A Delphi technique survey was used to create a screening tool based on the experiences and expertise of a multi-disciplinary team of clinicians working in the field of PCa. The study explored the questions that should be included in a screening tool to investigate the NSSE after PCa as well as the appropriate wording of statements to include in the NSSE screening tool.

Methods

Study design

A modified Delphi study was performed and outlined using the methodological criteria to report in publications as set out by Diamond et al (2014) (22), and all methods were carried out in accordance with these guidelines. The purpose of the Delphi study was to obtain consensus among experts on the questions to include in a screening tool for NSSE after PCa treatment. Three rounds of the study survey were circulated (23). The potential participants were emailed a study recruitment letter along with a Google Forms link to participate in the first of the three predetermined rounds of the study. The recruitment letter informed them of the study and explained all aspects related to the study. The duration of the study was predetermined (22), and was set as three rounds each consisting of 3 weeks, with a one-week collation time after each round, making the total duration of the study 12 weeks. The time to complete each round was suggested to take only 10–15 minutes. Participants were assured anonymity and informed of their right to withdraw at any time. All the participating experts gave informed consent to participate in the study. Consensus was defined, and the termination of the study was described. Each participant was asked to complete the study survey independently and were given instructions on how to complete each round of the study.

The first round (RD 1) included demographic questions on the expert panel. All three rounds (RD1-3) presented a set of statements in the form of questions to be posed to a potential patient. The experts were asked to indicate how appropriate they thought the statement was by ranking it on a 5-point Likert scale (“strongly agree, agree, neutral, disagree and strongly disagree”) Participants were also asked to submit comments and suggestions during each round to improve each statement and were given unlimited space for this purpose. This allowed for the identification of statements that were unclear or required additional attention. Once a participant submitted their survey answers, the study moderator was able to collate their information and code each participants’ data into an Excel spreadsheet. Participants who had not yet responded during each round were received two additional reminders to complete the round, and the Google form link was closed after three weeks. The research team discussed and implemented all the comments and suggestions and communicated the changes and the new version of the screening tool to the experts during subsequent rounds. The experts were thus asked to rank the appropriateness of a new set of statements in RD2 an RD3 according to the changes that the collective group of experts requested in the previous rounds.

Results

This section presents the results of the Delphi process and how consensus and stability evolved over the RD1-RD3 by looking at:

• The composition of the expert panel

• Agreement percentages

• The evolution of statements

• The support of statements by the expert panel.

Thirty-five participants were initially invited to participate in the study, and 27 responded and completed round one, 23 participants responded and completed round two, and 20 participants completed round three (Fig. 1). The 3 rounds were successfully executed in the planned 12-week time frame.

Participant demographics: RD 1–3

Table 1 describes the detailed demographic information of the participants during each round. The largest professional representation of the experts during round 1 were the combined group of urologists (30%), followed by the combined group of sexologists (26%) followed by the physiotherapists (22%). This trend remained consistent during round 2 except for the sexologists making up the majority of the group at 30%. This was similar in round 3 with the sexologists constituting 30% of the expert group and the urologists and physiotherapists each representing 25%.

The ratio of male to female participants were equally split throughout the rounds. Two thirds of the participants had a minimum of 15 years’ or more experience in the field. The overwhelming majority of the participants were qualified at Masters level or PhD throughout the rounds (R1: 78%, R2: 83% R3: 80%). Most participants were practicing in the private sector (74%).

Statement agreement between participants: RD 1–3

The agreement for each statement for Rd1-3 is presented in Table 2. The results include the total responses received for each round, and percentage breakdown between the strongly agree, agree, neutral, disagree and strongly disagree options.

Statement 1

Please refer to box 1.

Box 1 : Agreement, statement support and statement evolution for statement 1

Round 1

The word “loss” and “arousal” was found to be problematic and replaced with “leaking” and “arousal (besides during an orgasm)”. A definition of arousal was suggested and included in round 2. Adjustments were proposed related to the Likert scale that was used, and the “never to always” scale was replaced with a “very rarely to very frequently” scale. One expert (a urologist) stated that this was not a side effect, especially not after brachytherapy.

Round 2

It was suggested that “with or without a partner” and “with or without an erection” needed to be added to the definition of arousal. Consensus was reached.

Statement 2

Please refer to box 2

Box 2: Agreement, statement support and evolution of statement 2

Round 1

It was suggested that “with your orgasm” be replaced with “during an orgasm”, “loss” to be replaced with “leaking”. There was confusion between “orgasm” and “ejaculation”,

Round 2

The experts asked that a statement needed to be added that an orgasm may occur with or without ejaculation. Consensus was reached.

Statement 3

Please refer to box 3

Box 3: Agreement, statement support and evolution of statement 3

Round 1

In relation to the wording some experts thought that the statement implied that an orgasm was already being achieved. The first part of this statement was subsequently changed to establish whether an orgasm was being achieved. Other suggestions required an amendment to the response on the Likert scale by changing the wording from a “decrease to increase scale to “much less to much more scale”.

Round 2

It was suggested to swap the order of statement 2 and 3 to improve the flow of questioning. This was implemented in the final round. Consensus was reached.

Statement 4

Please refer to box 4

Box 4:

Round 1

Suggestions were made to add a description of the area of symptoms, and to allow a way to quantify/measure the pain on a scale. This question was elaborated in round 2 to include frequency value to how often symptoms occur, a measuring capacity using the numeric pain rating scale (NPRS) and the allowance for descriptive words in the answers to allow for more detail on area of symptoms and descriptions of symptoms.

Round 2

A suggestion was made to replace the NPRS with the visual analogue scale. This was rejected due to the fact that sizing of the scale may change on different screens/platforms losing its reliability.

Round 3

A comment was made to simply state “Have you… instead of how often have you “. Another comment was made that the description of the patient’s pain would not be valuable, as it could not be used to distinguish different types of pain.

Statement 5

Please refer to box 5

Box 5:

Round 1

The urologists on the panel expressed strong concern that this statement may be misleading to patients, as anejaculation is a given consequence for most post- prostatectomy patients. This was addressed in round 2, where the question was first asked whether ejaculation is able to occur. There were also concerns that some men may associate the ejaculation event as the actual orgasm event, and not be aware that an orgasm is possible without ejaculating.

Round 2

A comment was once again made whether prostatectomy patients would get confused, as they will not be able to ejaculate after their treatment. It was thought that the question may confuse patients and that it may leave patients concerned that their surgery was performed poorly/incorrectly.

Round 3

Suggestions were made to remove sections of the question. Some experts also expressed that it would be inappropriate to ask about a change in volume of ejaculate.

Statement 6

Please refer to box 6

Box 6:

Round 1

A suggestion was made to include a section for other options that were not mentioned. The word “asleep” was queried, and suggested to be changed to “numb”, which was done.

Round 2

A grammar comment was made relating to the Likert scale and implemented in round 3.

Round 3

Suggestions were made to add the “how problematic” section to this question, similar to some of the other statement. These adjustments were made. Consensus was reached.

Statement 7

Please refer to box 7

Box 7:

Round 1

Suggestions were made to remove the options of how much the decrease in size was estimated at, and to keep the question more general. There were suggestions to change the word “bothersome” which was done in round 2.

Round 2

Some comments were made related to the impact of the penile shortening on self-confidence and self-image, but these were not considered for this questionnaire.

Round 3

Suggestions were made to add a time scale and the partners’ perspective to the question. These suggestions were not considered as the partners perspective was already invited at the start of the questionnaire, and the time scale was already included for referencing purposes.

Statement 8

Please refer to box 8

Box 8:

Round 1

A suggestion was made to change the phrase “different curvature” to “any new curvatures”, as some minor penile curves were deemed normal. One suggestion from a urologist was to remove this question as it was not a known consequence. The same suggestions that were made to change “bothersome” in statement 7 were again made, and changes were implemented in round 2.

Round 2

A statement was made by a urologist that this question does not belong as it does not occur with cancer treatment. A comment was made to replace the word “curvature” with “change in shape.” This was not considered for the final round. Another comment suggested to include the word “bend” along with “curvature”, this was included in the final round.

Round 3

Similar comments were made to statement 7 regarding the partners perspective and time scale. A urologist on the expert panel stated that this side effect was not a consequence of PCa treatment. There was also again a suggestion to include “shape” in this question. This was not included as shape was seen as a misleading inclusion as it could mean many different things.

Statement 9

Please refer to box 9

Box 9:

Round 1

Suggestions were made to split the 2 questions completely. It was also suggested to include “intimacy” with the phrase. there were many positive comments regarding the fact that this was an open-ended question, and this would give context to the symptoms.

Round 2

Most panellist agreed that this was an important question, but its appropriateness for inclusion in this quantitative questionnaire was questioned. Comparisons were made with other similar questionnaires that did not have open ended questions. This question was therefore completely removed from the questionnaire.

Statement 10

Please refer to box 10

Box 10:

Round 1

It was suggested that the statement be split into two statements, or be rephrased as the question seemed a bit wordy.

Round 2

As with statement 9, most panellist agreed that this was an important question, but its appropriateness for inclusion in this quantitative questionnaire was questioned. Comparisons were made with other similar questionnaires that did not have open ended questions. This question was therefore removed from the questionnaire.

Discussion

This Delphi study was conducted to establish agreement on the statement and wording to be used on a new self-administered screening tool that aims to initiate a discussion on the NSSE between a clinician and a patient after their prostate cancer treatment. The purpose of this Delphi study was to enable an expert panel to agree and reach consensus on the composition and wording of a screening tool that could be used to identify the Neglected Sexual Side effects after PCa treatment. A modified Delphi study has a predetermined number of rounds, and in this case, it was decided to conclude after 3 rounds, if consensus has still not been reached (23).

The experts on the panel included a multidisciplinary team of whom all but one, a medical sexologist, were based in South Africa. Most of these experts were in private practice, which is where the majority of early stage prostate cancer patients are managed due to resources limitations in the government run health sector (4, 28).

The experts in this study were experienced and covered a multi-disciplinary group of clinicians involved in PCa treatment. Each group of experts displayed specific areas of interest within the scope of the screening tool being developed. The urologists on the panel expressed concern about the definition and wording of statement 5 (round 1) that relates to “anejaculation”. They expressed the need for unambiguity in stating that anejaculation was a given consequence after a prostatectomy and not a side effect of PCa treatment. The sexologists were very interested in the details of each NSSE and pushed for additional descriptions to be included to further expand on the sexuality aspects that may be impacted on. The psychologists were concerned with the impact that the NSSE would have, and the view of the patient’s partner. The physiotherapists and oncologists had more general comments throughout the study.

Stability of consensus in this study was measured using agreement percentages and statement support parameters. Statement 2 was the most stable statement, as it had the smallest mean (1,40 and 1,09), smallest standard deviation (0,61 and 0,29) and the lowest range (1–2 and 1–2) between rounds. Statements 1 and 3 were also stable and were well supported by panellists with improvements made from round 1 to 2, and reaching consensus in round 2. While statements 4–8 all had increased in stability from round 1 to 2, they weakened from round 2 to 3 with regards to statement support from the panellists. Most comments and deliberations were made on these statements. Statements 4–8 all improved in their agreement over the 3 rounds. Statement 9 and 10 both had good stability but were removed after round 2.

Statement 5 consistently had the poorest range of statement support due to an outlier opinion of one panellist. This was due to the deliberation on the word “ejaculation” within this statement. Similar outlier opinions were noted in round 3 of statement 8, weakening the statement support in round 3 for the statement. In statement 8, one expert (urologist), repeatedly requested the removal of the Peyronies disease/penile curvature statement and argued that the disease was not a known side effect after PCa treatment. Published literature relating to Peyronies disease, however showed the presence of an abnormal penile curvature in 10% of participants in a 2014 study (14), and in 12% of the participants in a 2017 study after radiation treatment(12). This statement was retained as part of the screening tool for statement 8.

Ultimately, the experts reached 75% agreement/or disagreement on 4 of the statements, and majority agreement as per scenario 2 were reached on 4 statements. Two statements were removed and the final screening tool, in the form of a questionnaire, consisting of 8 statements was created.

Final screening tool considerations:

General suggestions included the desire for the questionnaire to remain brief and uncomplicated and this was implemented in the final questionnaire. Suggestions were made to remove the subjective options describing each of the side effects, and to focus on the impact it had on sexual activity, as was the case in the initial phrasing of statements 7 and 8. These were carefully considered and subsequently implemented. All the statements in round 1 ended with “during the last 3 months”. This phrase was removed from each individual statements in round 2 and included as an instruction for patients to “think about the last 3 months and compare that to the time before your prostate cancer treatment, and then answer the question”. The final screening tool produced is outlined in Table 3.

Conclusions

This study add value in that it will assist health care practitioners to identify a variety of sexual dysfunction complications, collectively referred to as “Neglected Sexual Side Effects in men after PCa treatment which are currently undiagnosed/ and not treated especially in LMIC such as SA. A variety of expert panellists representing related fields and with higher qualifications served as panel experts. Consensus was reached on the statements making up the NSSE screening tool. This screening tool may be applied on patients who have had treatment for early stage PCA that includes prostate surgery and prostate radiation therapies. This screening tool will need to undergo further psychometric testing to establish its reliability as well as its appropriateness for use in different settings.

Abbreviations

PCa: Prostate Cancer, NSSE:  Neglected Sexual Side Effects, Expanded Prostate Cancer Index, IIEF: International Index of Erectile Function.

Declarations

Ethics approval and consent to participate

Full ethical clearance was obtained from the University of KwaZulu-Natal, School of Health Sciences Research Committee (Biomedical Research Ethics Committee) with registration no: BREC/00000478/2019. All the participating experts gave informed consent to participate in the study.

Consent for publication

All participants consented that the data produced from this study would be published.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

None

Funding

None

Author contributions

All authors conceived and designed the study. All authors were involved in drafting and finalising the manuscript.  KN and JM provided additional clinical input for the study, and all authors approved the final version to be published. 

Acknowledgements

The authors would like to thank the panel of experts for their contributions to this study.

Authors information

Pierre Röscher: Currently a PhD student in the discipline Urology, at Nelson R Mandela School of Medicine, University of KwaZulu Natal, South Africa

Dr. Kimesh Naidoo is a Paediatrician and he is a lecturer and an affiliate to the Nelson R Mandela School of Medicine, University of KwaZulu Natal, South Africa

Dr. Joanne Milios is a men’s health physiotherapist and clinical researcher affiliated to the School of Sport Science, Exercise & Health, The University of Western Australia.

Prof. Jacqueline van Wyk is an associate professor at the Nelson R Mandela School of Medicine, University of KwaZulu Natal, South Africa

References

1. Salonia A, Adaikan G, Buvat J, Carrier S, El-Meliegy A, Hatzimouratidis K, et al. Sexual Rehabilitation After Treatment for Prostate Cancer-Part 1: Recommendations From the Fourth International Consultation for Sexual Medicine (ICSM 2015). The journal of sexual medicine. 2017;14(3):285-96.
2. Albaugh JA, Sufrin N, Lapin BR, Petkewicz J, Tenfelde S. Life after prostate cancer treatment: a mixed methods study of the experiences of men with sexual dysfunction and their partners. BMC Urol. 2017;17(1):45.
3. Global Burden of Disease Cancer C, Fitzmaurice C, Abate D, Abbasi N, Abbastabar H, Abd-Allah F, et al. Global, Regional, and National Cancer Incidence, Mortality, Years of Life Lost, Years Lived With Disability, and Disability-Adjusted Life-Years for 29 Cancer Groups, 1990 to 2017: A Systematic Analysis for the Global Burden of Disease Study. JAMA Oncol. 2019;5(12):1749-68.
4. Babb C, Urban M, Kielkowski D, Kellett P. Prostate cancer in South Africa: pathology based national cancer registry data (1986-2006) and mortality rates (1997-2009). Prostate Cancer. 2014;2014:419801.
5. Heyns CF, Fisher M, Lecuona A, van der Merwe A. Prostate cancer among different racial groups in the Western Cape: presenting features and management. S Afr Med J. 2011;101(4):267-70.
6. Heidenreich A, Aus G, Bolla M, Joniau S, Matveev VB, Schmid HP, et al. EAU guidelines on prostate cancer. Eur Urol. 2008;53(1):68-80.
7. Bamidele O, McGarvey H, Lagan BM, Ali N, Chinegwundoh Mbe F, Parahoo K, et al. Life after prostate cancer: A systematic literature review and thematic synthesis of the post-treatment experiences of Black African and Black Caribbean men. Eur J Cancer Care (Engl). 2018;27(1).
8. Bessaoud F, Orsini M, Iborra F, Rebillard X, Faix A, Soulier M, et al. [Urinary incontinence and sexual dysfunction after treatment of localized prostate cancer: Results from a population aged less than 65years old]. Bull Cancer. 2016;103(10):829-40.
9. Sanda MG, Dunn RL, Michalski J, Sandler HM, Northouse L, Hembroff L, et al. Quality of life and satisfaction with outcome among prostate-cancer survivors. N Engl J Med. 2008;358(12):1250-61.
10. Ben Charif A, Bouhnik AD, Courbiere B, Rey D, Preau M, Bendiane MK, et al. Patient Discussion About Sexual Health With Health Care Providers After Cancer-A National Survey. The journal of sexual medicine. 2016;13(11):1686-94.
11. Salonia A, Adaikan G, Buvat J, Carrier S, El-Meliegy A, Hatzimouratidis K, et al. Sexual Rehabilitation After Treatment For Prostate Cancer-Part 2: Recommendations From the Fourth International Consultation for Sexual Medicine (ICSM 2015). The journal of sexual medicine. 2017;14(3):297-315.
12. Frey A, Pedersen C, Lindberg H, Bisbjerg R, Sonksen J, Fode M. Prevalence and Predicting Factors for Commonly Neglected Sexual Side Effects to External-Beam Radiation Therapy for Prostate Cancer. The journal of sexual medicine. 2017;14(4):558-65.
13. Frey AU, Sonksen J, Fode M. Neglected side effects after radical prostatectomy: a systematic review. The journal of sexual medicine. 2014;11(2):374-85.
14. Frey A, Sonksen J, Jakobsen H, Fode M. Prevalence and predicting factors for commonly neglected sexual side effects to radical prostatectomies: results from a cross-sectional questionnaire-based study. The journal of sexual medicine. 2014;11(9):2318-26.
15. Nelson CJ, Choi JM, Mulhall JP, Roth AJ. Determinants of sexual satisfaction in men with prostate cancer. The journal of sexual medicine. 2007;4(5):1422-7.
16. Roberts KJ, Lepore SJ, Hanlon AL, Helgeson V. Genitourinary functioning and depressive symptoms over time in younger versus older men treated for prostate cancer. Ann Behav Med. 2010;40(3):275-83.
17. Nelson CJ, Mulhall JP, Roth AJ. The association between erectile dysfunction and depressive symptoms in men treated for prostate cancer. The journal of sexual medicine. 2011;8(2):560-6.
18. Szymanski KM, Wei JT, Dunn RL, Sanda MG. Development and validation of an abbreviated version of the expanded prostate cancer index composite instrument for measuring health-related quality of life among prostate cancer survivors. Urology. 2010;76(5):1245-50.
19. Rosen RC, Riley A, Wagner G, Osterloh IH, Kirkpatrick J, Mishra A. The international index of erectile function (IIEF): a multidimensional scale for assessment of erectile dysfunction. Urology. 1997;49(6):822-30.
20. Rowe GW, George & Bolger, Fergus. The Delphi Technique: A Re-Evaluation of Research and Theory. ORG Group & Teams (Topic). 2011.
21. de Villiers MR, de Villiers PJT, Kent AP. The Delphi technique in health sciences education research. Medical Teacher. 2009;27(7):639-43.
22. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401-9.
23. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393.
24. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758-72.
25. Hsu CS, BA. The Delphi Technique - Making Sense of Consensus. Practical Assessment Research adn Evaluation. 2007;12(10).

26. Humphrey-Murto S, Varpio L, Wood TJ, Gonsalves C, Ufholz LA, Mascioli K, et al. The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review. Acad Med. 2017;92(10):1491-8.
27. Schneider P, Evaniew N, Rendon JS, McKay P, Randall RL, Turcotte R, et al. Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology. BMJ Open. 2016;6(5):e011780.
28. Le Roux HA UR, Sartorius B, Aldous C. Prostate Cancer at a regional hospital in South Africa: we are only seeing the tip of the iceberg. South African Journal of Surgery. 2015;53:57-62.