Steering Committee
The Steering committee will include 5 members, including 2 medical oncologists in the field of LC (1 TCM expert and 1 western medicine expert), 1 methodologist, 1 policy maker and 1 COS developer. The committee will review and provide guidance at each stage of the study protocol.
Working group constitution
The working group will be made up of 10 individuals, including 4 professionals, 2 methodologists, 2 PhD students from the team of COS-TCM-LC development from Chinese clinical trial core outcome sets research center(ChiCOS) and 1 patient representative. The role of the working group will include organizing regular meetings, facilitating communication and hold discussion meetings to seek advice from the steering committee if there are any differences to be resolved.
Patient and public involvement (PPI)
Patients or their representatives will be involved in the semi-structured interviews, two rounds of Delphi survey and the final consensus meeting.
Design
The development of this COS-TCM-LC will be conducted in 4 stages (Fig.1):
-
Identifying a preliminary checklist of outcomes, after constructing a pool of outcomes and classifying into outcome domains (Stage 1).
-
Choosing stakeholder groups to be involved (Stage 2).
-
Conducting a two-round Delphi survey (Stage 3).
-
Holding a consensus meeting to determine final COS-TCM-LC (Stage 4).
Stage 1: Identifying a preliminary checklist of outcomes
The first step in the development of COS-TCM-LC is to collect all existing outcomes to form an outcome pool, which is an initial list of potential outcomes retrieved from TCM related clinical trials in LC. An outcome pool for TCM related clinical trials refers to a collection of outcomes that have been used in clinical trials concerning a certain disease or specific disease type with application of TCM related interventions. The outcome pool can generate a preliminary list of outcomes which is used in Delphi process in the development of a COS. In this study, we use four steps which include a systematic review of published literature, a systematic review of clinical trial registration protocols, clinician questionnaire surveys and patient semi-structured interviews to generate a pool of outcomes in identifying the preliminary checklist of outcomes.
Step 1: Systematic review
To identify outcomes used previously in clinical trials, we will perform a systematic review of randomized controlled trials (RCTs) for TCM.
Literature search and screening
A comprehensive, electronic search strategy of 3 English databases which include PubMed, Cochrane Library, and Embase as well as 3 Chinese databases which include the China National Knowledge Infrastructure(CNKI), WanFang Database and SinoMed. The search time range will be from 1 January 2010 to 31 December 2020. Key terms used to guide the search will include ‘lung cancer’, ‘small cell lung cancer’, ‘Large cell lung cancer’, ‘fei ji’, ‘clinical trial’ and ‘randomized controlled trial’. Two reviewers will independently screen the abstracts and titles of studies identified in the literature search. Relevant studies will be selected according to the prespecified inclusion and exclusion criteria as listed in Table 1. Any disagreements will be resolved through discussion after a thorough reading of the paper or by consulting a third researcher.
Table 1 Inclusion and exclusion criteria of studies
Inclusion criteria
|
Exclusion criteria
|
1. Study type: Only RCTs will be selected
2. Target population: Adult patients with clear diagnosis of LC
3. Intervention: TCM related treatments, which includes oral or intravenous Chinese medicine and Chinese patent medicine
4. Outcomes: Retrieval of all outcomes in the included study
|
1. Studies which involve other diseases
2. Studies on acupuncture, Tuina and acupoint application as intervention measures
3. Animal or experimental research, review, dissertation, conference paper
4. No description of LC outcome measures
5. Full-text which cannot be obtained.
|
Data extraction
Two reviewers will independently extract relevant data by reading the full texts from included studies. Data extraction will include the first author’s name, number of recruited patients, patients’ age, TCM syndrome types, intervention details (medication name, course of treatment, treatment frequency and dosage), name or definition of outcomes, outcome measurement instruments or methods, time-point at which the outcome is being measured. In addition, we will assess the quality of outcome reporting. As there is currently no consensus method for assessing the quality of outcome reporting, we refer to relevant COS development studies that have utilized the quality assessment of outcome measures reported[24-26]. The list of criteria to assess the reporting quality of the outcome measures is shown in Table 2. If the eligible study completely meets an item, it will receive 1 point. If it does not meet any item, it will receive 0 points.
Table 2 Seven items assessment of the reporting quality of outcome measures
No.
|
Criterion
|
Yes
|
No
|
1
|
Whether the outcome was clearly stated as primary or secondary outcome.
|
1 point
|
0 point
|
2
|
Whether the outcome was defined or not. Outcomes were considered defined if text of their meaning or a citation was provided.
|
1 point
|
0 point
|
3
|
It was clearly described how the outcomes are measured or the outcome measurement (indicators and/or tools used, if relevant).
|
1 point
|
0 point
|
4
|
It was clearly described by whom the outcomes are measured.
|
1 point
|
0 point
|
5
|
It was clearly described the time points and time period at or during which outcome was measured.
|
1 point
|
0 point
|
6
|
Are methods used to enhance the quality of outcome measurement (for example, repeated measurement, training) if appropriate?
|
1 point
|
0 point
|
7
|
The reporting of outcomes was consistent throughout the article. There is no unambiguous reporting that makes it confusing for the reader to assess what has been done.
|
1 point
|
0 point
|
Step 2: Identifying additional potential outcomes from 2 clinical trial registry databases
Two clinical trial registry databases, Chinese Clinical Trial Registry (ChiCTR) and ClinicalTrials.gov will be searched to include all RCTs on LC using TCM as an intervention to retrieve potential outcomes. The retrieval time is from inception to December 31, 2020. Two reviewers will independently screen the abstract and title of the studies identified in the clinical trial registry databases. If a study has been registered, completed or published, we will prioritize inclusion of the published full text to avoid duplication with published literature retrieved in Step 1. Relevant articles will be selected according to the inclusion and exclusion criteria shown in Table 1. The reviewers will independently extract relevant data by reading the full texts from included studies to determine its eligibility. Any disagreements will be resolved through discussion after a thorough reading of the paper or by consulting a third researcher.
An extraction table will be predesigned by the working group to collect data from the included studies. Data to be extracted will include country of registered organization/researcher, status of registered trial (ongoing or completed), ethics, funding resources, mode of intervention/s, name or definition of outcomes, outcome measurement instruments or methods and the time-point at which the outcome is being measured.
Step 3: Patient semi-structured interviews and clinician questionnaire surveys
Patients and clinicians will be invited to participate in semi-structured interviews and questionnaire surveys respectively. Recruitment will be carried at the respiratory and pulmonary department of inpatient wards and outpatient clinics in Tianjin University of TCM (TJUTCM) affiliated First Hospital and TJUTCM affiliated Second Hospital. The inclusion and exclusion criteria for the patient participants and healthcare professionals are shown in Table 3.
The recruitment sample size of patients and healthcare professionals will be 25 respectively. Healthcare professionals panel will include 13 TCM doctors and 12 western medicine doctors. We will solicit and respect the legitimate rights and interests of these subjects. If they agree to participate in the semi-structured interviews, we will ask them to provide a signed written consent form. In addition, they can withdraw unconditionally from the study without affecting any other interests.
At the beginning of the interview, a qualified and trained researcher of the working group will explain the study in plain language to prepare patients or their caregivers for the interview. The demographic and LC disease information of patients will be collected. At the same time, the trained investigator will ask the patient two questions to obtain the most important outcomes reported by patients (PRO) on LC. Examples of the questions during the interview are as listed below:
-
“what is the most important outcomes for you in the treatment of LC?”
-
“In the treatment of your LC, what aspects do you most hope to get better improvement?”
If the participants are not able to answer the questions, the investigator will provide the list of outcomes collected from the systematic review of published literature and registered studies for them to choose. The content of this semi-structured interview will be audio recorded.
For healthcare professionals, we allow them to have the freedom to list outcomes which are of importance to them for TCM treatment in LC using a questionnaire in an open-ended format. Participants will not be constrained by a definite list of outcomes to choose from. Participants are required to list up to a maximum of 5 outcomes. The limit set on the selection number can help to achieve the selection of outcomes which are of high importance to healthcare professionals.
Table 3 Inclusion and exclusion criteria for participants in qualitative studies
|
Inclusion criteria
|
Exclusion criteria
|
Patients
|
①LC patients previously or currently under TCM treatment
② Aged from 18 to 70
③ Have signed the informed consent
④Have adequate language communication skills
|
① Patients in acute stage
② Patients with severe mental illness
|
Healthcare professionals
|
① Hold a postgraduate degree and above
|
|
②Specializing in thoracic surgery or medical oncology
|
③ Have signed the informed consent
|
Step 4: Constructing an outcome pool
Outcomes collected from systematic reviews, clinical trials registry databases and qualitative surveys will be combined to form a pool of outcomes. An initial translation of audio-recorded qualitative surveys into texts will be performed and outcomes retrieved will be imported into an Excel table. This will be followed by different names of a similar outcome will be standardized and unified as one, which is done in accordance to the definition of the outcomes in original studies or videos of the qualitative surveys. Any duplicates will be removed from the outcome list. Similar outcomes with overlapping definitions will be merged. Finally, an outcome pool of LC using TCM interventions will be developed. Outcomes will then be classified 7 outcome domains, namely TCM syndrome or symptom, clinical symptoms and signs, physiological and biochemical indexes, quality of life, long term prognosis, economic evaluation and adverse events. In this step, two researchers will independently standardize, merge and group the outcomes, and then cross-check the results after completion. If there is any disagreement,it will be resolved by discussion with a third researcher.
Step 5: Establishing a preliminary checklist of outcomes
After constructing a pool of outcomes, an initial list of outcomes will be formed. The initial list of outcomes can be a long list. Implementing a long list of outcomes will deter participants in the Delphi survey (in the later stage) from completing and thus affect the response rate in the Delphi survey[21]. Thus, we need to reduce the length of the list when considering the practicality of the study. The preliminary checklist of outcomes will be determined based on the following (Fig.2):
-
If there are more than 80 outcome items in the pool of outcomes, specific criteria will be applied to remove some of the outcomes from the pool of outcomes. The outcomes will be assessed according to the 6 principles, which include the clinical importance of the outcomes, the combination of disease and symptom, the recognizability of the outcome, the specificity, consistency and practicality of the outcome in the study. If at least 90% of the members voted for an outcome as “not important”, the outcome item will be dropped from the list, with guidance by the Steering committee in an internal anonymous voting process. Finally, the retained outcomes will be approved by the Steering committee as the preliminary checklist of outcomes for COS-TCM-LC.
-
If there are 80 or less outcomes in the pool, all of the outcomes can be retained as items in the preliminary checklist.
Stage 2: Selection of stakeholder groups
The selection of stakeholder groups is crucial in the process of COS development[27-29]. Representatives of the stakeholder groups will determine the representation of a COS. In the study, the stakeholder groups include COS users, healthcare professionals (medical doctors and TCM doctors), clinical trialists, representatives of pharmaceutical enterprises, health policy makers, researchers, methodologists, patient representatives, COS developers and journal editors. A list of all potential stakeholders is shown in Box 1.
Box1 Stakeholder involvement
Healthcare professionals
➢ Have at least 2 years of clinical experience in LC
—Western doctors
—TCM doctors
Patients and representatives (family, carer)
➢ Diagnosed with LC
➢ Aged from 18 to 70 years old
➢ Have received TCM treatment
➢ Family or carer of a patient with LC
COS users
➢ Clinical trialists
|
Stage 3: Two-round Delphi survey
The Delphi survey will comprise of two rounds of electronic questionnaires. Delphi round 1 will include the generation of scores from each outcome in the preliminary checklist. Additional outcomes which are deemed to be important, and previously not included in the preliminary checklist, can be added into the list of items by participants. The added outcome items will be cross-checked to determine if these items are new and unique. The newly added items will be confirmed and added to the list of outcome items in Delphi round 2. Feedback scores from the first round will be presented in Delphi round 2 and the scoring statistics from all participants in various stakeholder groups will be collected and analyzed.
Participants who have completed the first round will be invited to score all items in the first round as well as newly added items in Delphi round 2.
Delphi survey participants
There is currently no recommendations for the sample size of a Delphi survey. We aimed to include a range of stakeholders’ perspective and ensuring an equal representation of healthcare professionals and patients. Healthcare professionals will include a fair mix of western doctors and TCM doctors. We will also ensure that patients in the early and late stage of LC currently or previously on TCM medication will be represented in the Delphi survey. The stakeholders will be invited to participate in the Delphi survey on a voluntary basis. We will invite 200 representatives identified through certain academic organizations such as China Association of TCM.
Delphi survey process
An electronic mail will be sent out to all participants (excluding patient representatives), which includes a link to the questionnaire survey. Each round of Delphi survey should be completed within four weeks. The patient representatives will complete the survey in a face-to-face interview with the trained researcher from the working group during outpatient consultation under the supervision of their medical doctor or TCM doctor.
Delphi scoring
Participants will be asked to score each outcome according to the importance using a 9-point Likert scoring scale, where 1 to 3 is labeled as ‘not important’, 4 to 6 as ‘important but not critical’ and 7 to 9 as ‘critical’[30]. Participants will have an option of ‘unsure’ if they are unable to assess the importance of the outcome items.
Delphi round 1
Participants are required to score every outcome. Simultaneously, an open-ended question will be placed at the end of the questionnaire where participants can suggest additional outcomes which they think is necessary and important to be in COS-TCM-LC. After the first round of Delphi,the working group will proceed to calculate the number of respondents and the distribution of scores for each outcome item. Outcomes which are defined as ‘important’ (if at least 70% or more participants scored 4 to 9) will be retained. For the remaining outcomes, the working group will discuss and retain outcomes with controversial opinions. In addition, outcomes added will be cross-checked to confirm if they are new items. Newly added items will be retained. All of the retained outcomes will be included in Delphi round 2.
Delphi round 2
On completion of Delphi round 1, the following participant will automatically be included in round 2 of Delphi survey. In Delphi round 2, aggregated results of the number of respondents and distribution of scores, together with their own score for every outcome in round 1 will be presented to all participants. They are required to rescore all outcomes, and if the score of some outcome changed significantly, for example, a score of less than 4 (not important) to a score of 7 and above (critical), or vice versa, they will be required to provide a reason for the scoring changes.
After the second round, the working group will then proceed to generate statistics of the number of participants, distribution of scores and assessment of attrition bias. The changes in scoring between rounds will be examined and verified, as well as summarizing the reasons for the change in scores between rounds. The outcomes will be ranked according to the level of importance rating. Outcomes which have achieved consensus in at least one of the stakeholder groups will be included in the candidate list of core outcomes for the consensus meeting.
Stage 4: Consensus meeting
The objective of a consensus meeting is to determine the final core outcome items in a COS development study. A one-day face-to-face meeting of approximately 20 individuals will be held to discuss the results from the Delphi survey to agree upon a final COS. Apart from the Steering committee and the working group members, participants are mainly representatives of the stakeholder groups and senior experts in related fields who have completed 2 rounds of Delphi survey.
The meeting process will involve five sections:
(1) a short review of the COS study.
(2) summary of the scoring results of all outcomes from each stakeholder groups will be shown during the meeting, introducing all outcomes in the candidate list and obtaining the number of stakeholder groups which have achieved consensus.
(3) Participants will rate the candidate list of core outcome through anonymous voting.
(4) An outcome will be considered to have reached a consensus if 70% of participants vote in favour of the outcome to be included in the COS[31]. The determination of COS-TCM-LC will be via discussion of the voting results to achieve consensus. All participants have the rights to discuss any items, if there are any disputes, it will be settled via the guidance of the Steering committee. .
(5) Suggestions for later revision and promotion.
Consensus definition
Consensus definition is classified into 3 categories: ‘consensus in’, ‘consensus out’ and ‘no consensus’. These consensus criteria will be used for outcomes to be retained or dropped at the end of Delphi round 2. ‘Consensus in’ is defined if at least 70% of the participants in at least one stakeholder group score outcomes as 7 to 9, and will be given priority as a recommended item in the candidate list of core outcomes. ‘Consensus out’ is defined when more than 70% of all participants score outcomes as 1 to 3. ‘No consensus’ refers to outcomes which does not meet ‘consensus in’ and ‘consensus out’ requirements (Table 4). The selection of 70% for ‘consensus in’ relatively broadens the criteria for outcomes to be included in the candidate list of outcomes for the consensus meeting in determining the final COS-TCM-LC.
Table 4 Consensus definition
Consensus classification
|
Description
|
Definition
|
Consensus in
|
Consensus that outcome should be
included in COS
|
>70% score 7-9 in at least one stakeholder group
|
Consensus out
|
Consensus that outcome should not be included in the COS
|
>70% score 1-3 in all participants
|
No consensus
|
Uncertainty about importance of the outcome
|
Anything else
|