The COS development will follow recommendations of the Core Outcome Measures in Effectiveness Trials (COMET) initiative from the international Core Outcome Set Standards for Development (COS-STAD) (3, 4).
The Delphi method will be used to achieve a consensus from different stakeholders’ groups.
A prospective study protocol was registered in the COMET database (5). The protocol has been written following the Core Outcome Set-Standardised Protocol (COS-STAP) recommendations (6) in cooperation between Work Package 2 and 6.
In figure 1 each step of the study is illustrated.
Recruitment of participants mainly takes place from members of the HARMONY Work Packages, but also participants outside the HARMONY Alliance are invited to take part in the Delphi survey within their stakeholders’ group. At least three rounds of the Delphi survey are planned to achieve consensus. At the end a final face-to-face consensus meeting will take place.
Scope of the COS
The COS identified AML will be not only crucial for the HARMONY project but for all future analyses. As the definition of COS will determine what data are captured for AML, it will also impact the analyses will be available in the future. Consequently, COS will directly impact future data sets available to HARMONY and also the research community. Additionally, future work conducted by HARMONY partners and other stakeholders in the AML field will be directly impacted by the defined COS.
Every patient older than 18 years with AML can participate. Different subtypes of AML are equally included, regardless of previous treatments including stem cell transplantation. Patients treated as outpatients were included as well as patients treated in hospitals. Due to the use of English for the Delphi survey, participation is limited to patients understanding English.
- Clinicians and clinical researchers
Every clinician within or outside the HARMONY Alliance who has experiences in AML treatment can participate.
- Drug developers
Members of EFPIA and part of the HARMONY Alliance are invited to participate as well.
Recruitment of participants will be performed within the HARMONY Alliance with the support of Work Package 7, that is responsible for dissemination, communication and training within the HARMONY Alliance. Work Package 6 provides assistance as well in promotion for the survey.
To recruit health care professionals AML key opinion leaders will be contacted and will be asked to invite other professionals within their peer groups to take part in the Delphi survey. Thus, participants outside the HARMONY Alliance are welcome to take part in the Delphi survey. Patient recruitment will be performed with the support of patient advocates and several patients’ organisations, like Acute Leukemia Advocates Network or LeukaNet amongst others. Especially in recruitment of patients, social networks and internal communication tools will be used to spread the information and invitations for the Delphi survey. In the next rounds, personal mail reminder will be sent out to enhance the response rates.
Ethics approval and consent to participate
HARMONY Alliance is structured into eight work packages of which Work Package 8 (WP 8) is responsible for ethical aspects of the project. Further information can be found here:
In consultation with the WP 8 the development and conducting of the Delphi survey was performed and an ethics approval is not required.
Recruitment of participants is made within HARMONY Alliance. Especially patients’ recruitment is made by patient umbrella organisations, partners in HARMONY.
Before registration, all study participants receive information about the study and informed consent is obtained from all participants during the registration procedure.
This project has been registered (May 2019) in the database of the Core Outcome Measures in Effectiveness Trials (COMET) initiative.
Study management group
As recommended by the COMET initiative a study management group has been assembled to oversee the project (3). The group comprises a study coordinator, a haematologist with leading roles in AML treatment and clinical trials, drug developer with experience in past and current trials, patient advocates and methodological experts with experiences of systematic reviews and Delphi studies.
The role of the study management group is to support the development of the study protocol and to review the list of outcomes and the associated lay versions and descriptions.
Selection of the preliminary outcome list for AML
The empirical basis for identifying a long list of preliminary outcomes relevant in AML for the Delphi study so far has been threefold:
- Step: A literature research was conducted in the COMET database to get an overview of the outcomes of cancer already used in existing clinical trials. Therefor all relevant trials for outcome set definition in AML were collected to 17th October 2017 (7). The primary AML outcome list was generated by extracting outcomes from the COMET database research and review of published literature (8 - 15).
- Step: Several semi-structured interviews of clinical public and private key opinion leaders were conducted to assess the initial selection of the outcome parameters and additional outcomes were supplemented. This was followed by several face-to-face meetings to further expand and discuss the potential outcome list, including a multi-stakeholder group workshop and a further meeting with European AML key opinion leaders within the scope of HARMONY Alliance.
- Patient representatives and patients were consulted to include patients’ perspective. The preliminary outcome list was complemented by including additional outcomes and revising in accordance with patients’ comments.
On the basis of new scientific knowledge (16) and recommendations of the COMET initiative (3) the outcome list was reworked. Prognostic factors, e.g. age and gender were removed from the list. Tools of “how to measure” an outcome, e.g. quality-of-life questionnaires were also removed. Instead, AML concepts (outcomes related to patients’ perception on their symptoms, functioning and health-related quality-of-life) were included.
After the pre-Delphi stage a list of 59 outcomes, each with a short plain language description, grouped into 8 domain categories was completed. You can find the list used for this Delphi survey as additional file 1.
The Delphi survey will be managed online using DelphiManager software maintained by the COMET initiative (7). Invitations with a registration link will be sent out. After registration every participant will receive a unique identification code to take part of the Delphi survey. For registration, participants are asked to fill in their email-address, their stakeholder group and their home country. The webpage includes a description about aims and objectives of the survey and gives an explanation about how to complete the online Delphi survey. In every round the participants will be asked to rate importance of each outcome based on their personal experiences. Each outcome will be ranked into three categories (“not important”, “important, but not critical” and “critical”) using a 9-pointed Likert-scale.
After the first round a descriptive statistic for every stakeholder group will be provided. Only participants who completed the first round will be invited to take part in the second Delphi round. In the following rounds participants will revise their answers by taking the previous results into account. The process is stopped after pre-defined consensus criteria.
To reduce potential bias in interpretation of the results a clear consensus definition is important. We will use three categories of consensus that were already used in previous works (17).
- Consensus in = 70% or more over all respondents scored the outcome as critically important and 15% or fewer over all respondents scored the outcome as limited important
- Consensus out = 70% or more over all respondents scored the outcome as limited important and 15% or fewer over all respondents scored the outcome as critical important
- No Consensus
Outcomes that do not achieve consensus through several Delphi rounds will be discussed in a final face-to-face consensus meeting to finally ratify the AML core outcome set. Representatives from all participating stakeholder groups will be part of this meeting.
Analysis of the Delphi study will use descriptive statistics. The results for each Delphi round, for each outcome and for each stakeholder group will be presented in frequency tables. The analysis of the Delphi survey will be performed using the R statistical software version 3.5.2.
As an exploratory analysis we additionally identify outcomes considered as important for patients. The median Likert score for the patient group at the end of each round will be calculated and those outcomes achieving a median of greater or equal to 7 will be considered as important for patients and will be included in the COS. In this way patient-important outcomes can be separately discussed in the final consensus meeting. Attrition bias will be investigated by comparing results across participants who complete successive rounds versus those who withdraw at round 2 or 3.
Until now, no valuable data is available about the best group size for a Delphi survey, in this AML pilot Delphi a group size of 20 – 50 participants should be achieved.