Core outcome set development will follow recent guidance from the COMET initiative.46 A five-stage approach will be used (Figure 1). A comprehensive list of outcomes relevant to stakeholders will be compiled through a systematic review of outcomes reported in existing trials, supplemented by qualitative interview studies with patients and families, and clinicians. Second, the comprehensive list will be reduced to a core outcome set using Delphi methodology involving patients, families, and clinicians.
Third, to inform consensus discussions on inclusion criteria and outcome selection for future RCTs for acute severe paediatric asthma in the ED setting, we will review current clinical practice guidelines, existing clinical trials, and literature regarding bedside asthma severity scores. We will then identify practice differences in bedside assessment of asthma severity, and determine whether further prospective work is needed.
Fourth, a retrospective chart review in Australia and New Zealand will enable us to identify the incidence of serious clinical complications in children hospitalized with acute severe asthma, such as intubation, ICU admission, and death. Understanding the incidence of such outcomes will allow us to understand how common (and therefore how feasible) particular outcomes are in an asthma ED population.
Finally, a meeting of the PERN asthma working group will be held, with invitation of other clinicians interested in acute asthma research, and patients / families. Results of the initial steps outlined above will be presented, and the group will be asked to achieve consensus on a core set of outcomes and to make recommendations for the conduct of clinical trials on acute severe asthma. If this is not possible, the group will agree on a series of prioritized steps to achieve this aim.
Project 1: Developing a comprehensive list of outcome measures.
In order to develop a comprehensive list of outcome measures, we have performed a systematic review of outcomes reported in existing trials, which will be complemented by an Overview of Cochrane Reviews, and qualitative interview studies with patients and families, and clinicians.
We have recently conducted a systematic review of primary and secondary outcomes used in trials of IV bronchodilators in children with acute severe pediatric asthma. We searched MEDLINE, EMBASE, Cochrane CENTRAL, and the World Health Organization International Clinical Trials Registry Platform for randomized trials in children (less than 18 years) with acute severe pediatric asthma comparing IV bronchodilator therapy to another treatment.
Data extraction included authors, year of publication, geographic location, number of patients, primary, secondary and other outcomes used, and study interventions including dosing and timing of medication administered. The systematic review was prospectively registered in the PROSPERO database (CRD42017055331), and has recently been published.47 We identified 35 published papers and four registered study protocols. Fifty-six primary outcomes were identified, the most common being a clinical asthma score (23/56; 41%). Other identified primary outcomes included bedside tests of respiratory function (11/56; 20%) and measures of length of stay (9/56; 16%). There were a total of 60 different secondary outcomes, the most common were various length of stay measures (24/60; 40%) and adverse events (11/60; 18%).47
Overview of Cochrane Reviews.
Members of the PERN asthma working group are also undertaking an Overview of Cochrane Reviews to summarize Cochrane Reviews on the efficacy and safety of interventions for escalation of therapy for acute exacerbations of asthma in children. The protocol for this overview has been published.48 Additional objectives of the overview are to (1) identify gaps in the current evidence base that will inform recommendations for future research and subsequent Cochrane Reviews; and (2) to categorize information on reported outcome measures used in trials of escalation of treatment for acute exacerbations of asthma in children.
Qualitative interview studies: patients / families and clinicians.
This part of the project aims to assess the perspectives of patients, families and clinicians regarding their experience of acute severe paediatric asthma. Qualitative methods are recommended to identify outcomes important to stakeholders, help understand why these outcomes are important, and identify appropriate language to use when presenting these outcomes in later surveys.49 These perspectives will be combined with a similar study of clinicians, and the results of the systematic review to develop a comprehensive list of outcome measures. This list will directly inform the development of a planned Delphi survey (see below).
To ensure global representation and maximum diversity, we aim to analyse the perspectives from a number of patient and clinician groups from multiple hospitals in a variety of settings around the world. For patients and families, purposive sampling46 will occur, with selection of children with recent experience of acute severe asthma exacerbations requiring attendance in an emergency department and admission to hospital, managed in a variety of hospitals in a variety of health-care systems. For clinicians, we aim to include clinician groups from multiple EDs, paediatric wards, and ICUs. Initially, we will select clinicians from the already established asthma working group, and then – if further participants are required – expand to include intensive care physicians and hospitalists / paediatricians.
The specific objectives of the patient / family interviews are to: (1) determine which aspects of clinical care and which outcomes are most important to patients and families in this setting.; (2) determine the underlying reasons why these outcomes are important; and (3) identify possible participants for a patient steering group and participation in a consensus meeting. For the clinician interviews, our first two objectives are the same as the patient / family interviews, while a third objective is to determine how clinicians define an acute severe exacerbation of asthma.
Ethics approval for the interview studies has been obtained in Australia; additional local ethics approval will be sought where relevant. An overview of the methods of the interview studies for clinicians and patients / families is presented in Table 1.
Invitations to participate will be distributed via email through the partner networks that contribute to PERN, and to clinicians associated with the PERN asthma working group. Copies of the interview schedules are available as supplementary online material.
To provide a clinical context to each patient interview, a focused review of the relevant medical records for included patients will occur. We will extract details of the patient’s age, gender, previous asthma history, and treatment administered during the index hospitalisation.
Any identifying data will be removed from the interview transcript prior to analysis. Patient / family interviews conducted in a language other than English will be translated into English for the purpose of analysis. The translation will be checked by the original interviewer.
Thematic analysis will be performed independently by the principal investigator for each study (SC for the patient / family interview study and CG for the clinician interview study) and one other investigator, based upon the taxonomy developed for outcomes in medical research, comprising mortality/survival, physiological/clinical, life impact, resource use, and adverse events/effects.50 Additional taxonomy based on the Theoretical Domains Framework (TDF)51 will be used for the clinician interviews. The themes and coding will then be reviewed and cross-checked by the original interviewer. N-vivo qualitative analytic software (QSR International, Melbourne) will be used for analysis.
Due to the qualitative nature of the studies, we have not pre-determined a sample size. Table 1 provides details on how thematic saturation will be determined for each study. We are aiming to obtain representative views from diverse populations around the world, so anticipate the involvement of at least ten hospitals for each study.
Data management and storage
De-identified transcripts will be collected and stored in a password-protected file at Monash Health, Melbourne, Australia (patient interview study) and at Women and Children’s Hospital, Adelaide (clinician interview study). As per the National Health & Medical Research Council, we will keep original data for at least seven years.
The patient interview study has been approved by the Monash Health Human Research Ethics Committee (RES-18-0000-530A), whilst the clinician interview study has been approved by the Women’s and Children’s Health Network Human Research Ethics Committee (HREC/18/WCHN/120). Site-specific approval will be obtained at each site prior to commencement of local data collection.
Project 2: Prioritising outcome measures: Delphi process.
With information from systematic reviews and qualitative interviews, we will develop a comprehensive list of outcome measures. We will then conduct a prioritization exercise using modified Delphi methodology.
We intend to have two independent panels for the Delphi study: a patient / family panel and a clinician / researcher panel. Those participating in the qualitative research projects will be invited to participate in the Delphi study. Similar patient / family participants will be recruited from hospitals involved in the PERN member networks, and third-party asthma organizations, such as the National Asthma Council Australia, Asthma UK, Asthma and Allergy Foundation of America, and Asthma Canada. Additional clinician / researcher participants will be recruited through the PERN member networks and circulation of an invitation email by members of the PERN asthma working group to their local and national paediatric and/or emergency medicine organizations. Social media will also be utilized to recruit participants.
Due to the importance of minimizing attrition during the Delphi process,46 the requirement for participants to complete all rounds will be emphasized during recruitment, and reinforced during the informed consent process and ongoing communications (for example, using email reminders) once the study has commenced. The list of outcomes will be presented in plain, non-technical language. We intend to use an online survey platform such as Delphi Manager (http://cometinitiative.org/DelphiManager/) to conduct the study, and plan to create and circulate simultaneous surveys in English, Spanish and potentially other languages to allow participation of patients, families and clinicians based in the various countries contributing to the project. Reminders will be sent to non-responders at weekly intervals for two weeks for each round of the Delphi study. All questions and free-text responses will be professionally translated. Prior to commencement, we will seek approval from relevant human research ethics committees.
Delphi round one
In the first round, participants will be registered online, and provide basic demographic information. Clinician / researchers will be asked for details of their place of work and role (emergency physician, paediatrician, intensivist, etc), while patients / family members will be asked for details on their most recent hospital experience, and number of hospitalizations and ICU admissions. To enable identification of individuals completing each round, each participant will be allocated a unique identifier.
The comprehensive list of outcomes will be presented to participants in plain language. Questions will be grouped into topics (e.g. vital signs, length of stay measures, etc), and general questions will precede specific ones. Participants will be asked to rate the importance of each of the outcomes on a 9-point Likert scale, with 1 labeled as “not important”, and 9 labeled as “critical”. There will be space for participants to provide optional free-text comments for each choice. There will also be a free-text box at the end of the survey for respondents to suggest any additional outcomes not included in the questionnaire.
Delphi round two
Responses to the first Delphi round will be analyzed separately by the stakeholder group. No items will be removed. Open-ended responses will be collated, translated into English as necessary, and those suggested by two or more respondents will be added to the initial list.
The second round of the Delphi survey will present all items from round one as well as new open-ended responses. Items which were ranked in round one will be presented alongside feedback to participants with presentation of scores from both the patient / family group and the clinician / researcher group.
Delphi round three
Items for round three will be based upon the results of the second round, with inclusion if they are rated 7 to 9 (on the 9-point Likert scale) by 50% or more participants and 1 to 3 by no more than 15% of participants in at least one stakeholder group.46 Each item will be presented alongside round two scores from both stakeholder groups.
The survey for round three will also allow respondents to indicate their willingness to be contacted regarding a planned face-to-face meeting within the following 6-12 months to discuss the results of the study and determine a final list of outcome measures.
During all survey rounds, the questionnaires will be designed so that participants are not able to skip any questions or leave any questions blank, to ensure there is no missing data.
Definition of consensus.
The core set of outcomes from the Delphi survey will be selected after the third round of the survey according to the following criteria: (1) 70% of participants scoring outcomes as 7 to 9 and 15% or less scoring 1 to 3 by both stakeholder groups; or (2) 90% or more scoring 7 to 9 from either stakeholder group.46
Project 3: Reviewing current assessment tools for bedside assessment of asthma severity.
To understand current assessment practices, a review of currently used clinical guidelines and inclusion criteria for existing clinical trials is ongoing, along with a review of the literature regarding bedside asthma severity scores. The primary objective is to describe and compare acute severity assessment recommendations on childhood asthma exacerbations within and between geographic regions. This will be critical to ensure that patients enrolled in future studies of asthma treatments are comparable across regions. Secondary objectives are to compare management recommendations, and to assess the quality of existing guidelines, in order to clarify the extent to which current practice reflects the available evidence, and highlight areas for further research.
Sampling of clinical practice guidelines.
We will sample clinical practice guidelines by distributing emails inviting physicians and hospitals to participate via the eight partner networks that contribute to PERN, and to the clinicians associated with the PERN asthma working group. Email recipients will be requested to forward the email to other physicians and hospitals within their local geographic region and/or research network. The request for participation will also be shared on social media. This snowball approach, building on existing formal and informal professional and academic relationships, will allow sampling within countries without formal organised research networks.
Each participating hospital will be asked to provide copies and/or web links to their current clinical asthma guideline, and any regional or national clinical guideline.
Documents may be titled clinical protocols, clinical guidelines, care pathways, or other similar titles. However, for the purposes of this study, any document which provides recommendations on severity assessment and treatment for children aged >1 year of age presenting to the ED / hospital with acute onset of wheezing or asthma will be considered eligible for inclusion.
The clinical guideline project has been deemed a quality assurance activity by the Monash Health Human Research Ethics Committee (RES-18-0000-525Q), and had obtained details of over 110 local, national and international asthma guidelines by July 2019.
Data abstraction: guideline content.
Components of each clinical guideline will be independently abstracted by two reviewers. Abstracted data will be recorded on a purpose-designed password-protected spread-sheet.
If there are discrepancies between data abstraction between the two reviewers, discussion will occur. If discrepancies remain, a third reviewer will be consulted, and a decision will be made by consensus.
Guidelines which are written in languages other than English will be abstracted by two investigators who are fluent in English and the language in which they are written.
Specific data will be abstracted on:
- Definition of asthma (including age range)
- Assessment of acute asthma severity, including use of asthma severity scores. This will include abstraction of recommendations for when to utilise these scores (i.e. at triage, within the ED, while an inpatient), and whether the scores include clinical findings only, or additional investigations.
- Recommendations and thresholds for initiating treatments, including: (a) inhaled bronchodilator therapy; (b) enteral, parenteral, or inhaled corticosteroids; (c) adjunctive inhaled therapy; (d) IV / subcutaneous / intramuscular bronchodilator medications; (e) oxygen therapy (including devices and flow rates); (f) non-invasive or invasive ventilation; (g) admission / discharge; (h) ICU admission / interhospital transfer; (i) treatment at discharge; (j) follow-up management.
- Specific guidance for asthma complications (such as pneumothorax, atelectasis)
- Each recommendation will be abstracted in terms of direction (to use / not use), and, if available, strength of recommendation and quality of supporting evidence. Thresholds for treatment initiation will be abstracted according to history, examination, bedside tests of lung function, clinical asthma scores, and laboratory tests.
A copy of the data abstraction sheet is provided in the supplementary online material.
Comparisons will be made within each geographic region / research network, and across all guidelines. Specific comparisons will be made on treatment recommendations for severe asthma within each geographic region, including inhaled magnesium, IV beta-agonists, IV aminophylline, IV magnesium and non-invasive or invasive ventilation. Agreement between recommendations in the guidelines will be assessed.
Data analysis: guideline quality
Each clinical practice guideline will be assessed using the AGREE-II instrument.52 Each guideline will be independently assessed by at least two appraisers to increase the reliability of the instrument.52 All raters will undergo specific training on the use of the AGREE-II instrument. Guideline quality will be assessed overall and for each of the six domains examined by the AGREE-II instrument. Comparisons will be made between regions / research networks and all other guidelines, and between guidelines published prior to 2010 and those from 2010 and onwards.
Data management and storage
Data will be collected using a dedicated study email address, and extracted data stored in a password-protected electronic folder at Monash Health. Data will be stored for 7 years after the completion of the study.
Literature search and systematic review for asthma severity scores.
We will perform a systematic review to determine the reliability and validity of clinical bedside scores used for assessment of asthma and/or wheeze in children. Search terms will include [asthma OR wheeze] AND [validity OR validation OR reliability] AND [score OR scale OR assessment OR index], with appropriate modifications for each database, and be limited to children aged 0-18 years. We will also search for each clinical severity score identified by sampling of clinical practice guidelines. The search will be conducted in the Pubmed, Embase, Cochrane library, National Guideline Clearinghouse, and CINAHL databases. Each scoring system will be assessed with quality criteria relating to utility, reliability and validity.53
Step 4: Clinical outcome prevalence in acute severe paediatric asthma.
We will conduct a multi-centre retrospective cohort study of eligible children diagnosed with asthma or wheeze in 19 Australasian EDs associated with the Paediatric Research in Emergency Departments International Collaborative (PREDICT) network.
The study population will be children aged 1 to 17 years (up to but not including 18th birthday) presenting to a participating ED with a discharge diagnosis of asthma / wheeze who are administered inhaled or IV medication, or receive invasive or non-invasive respiratory support. We will exclude children with a final ED diagnosis of bronchiolitis and no bronchodilator administered, those with a diagnosis of foreign body inhalation, and those receiving IV bronchodilator treatment before arrival.
Each site investigator will identify a list of medical records of all children meeting inclusion criteria presenting between 1st November 2015 and 31st October 2016. These records will be identified using the local ED information system and hospital medical records, using the following ICD-10 codes: “Asthma” or “Childhood asthma” (ICD-10 code: J45, J46); “Wheezing” (ICD-10 code: R06.2); and “Acute bronchiolitis” (ICD-10 code: J21).
Each ED presentation is eligible for inclusion, even if the child has previously been included in the study. Each record will be assigned consecutive numbers by the medical records department, and then be screened for eligibility. Required information will be obtained from charts by trained abstractors.
The main outcome measure is the prevalence of IV bronchodilator therapy administration. Secondary outcome measures include type of bronchodilator therapy used; non-invasive respiratory support (CPAP / BiPap, and high-flow nasal oxygen therapy); prevalence and duration of intubation and mechanical ventilation; rates of hospital admission, ICU admission and interhospital transfers; complication rates for pneumothorax, hypokalaemia requiring replacement therapy, hypotension requiring an IV fluid bolus of at least 10 mL/kg, and arrhythmias requiring ECG monitoring or antiarrhythmic therapy.
Data storage and analysis.
De-identified data will entered into a protected electronic database at each participating hospital using REDCap (Research Electronic Data Capture)54, and be housed at a central data processing site at Murdoch Children’s Research Institute (MCRI), Parkville, Victoria. All study investigators and a biostatistician will have full access to final study dataset. The complete data set will be analysed at Monash Children’s Hospital, Clayton, Victoria.
Data will be descriptively analysed. Binary outcomes will be presented as proportions with 95% confidence intervals. For continuous outcomes, data will be presented as means and standard deviations (normally distributed data) or medians and interquartile ranges (non-parametric data).
For categorical data, comparisons will be made with calculation of odds ratio, and significance determined using Chi-square analysis. For non-parametric continuous data, comparisons will be made using the Mann-Whitney U test (2-group comparisons) and the Kruskall-Wallis test (3-group comparisons). For normally distributed continuous data, 2-group comparisons will be made using independent samples t-test, while 3-group comparisons will be made using ANOVA.
Exploratory modeling analysis
Multiple logistic regression will be used to assess predictors of ICU admission and other binary outcomes, while multiple linear regression will be used to assess continuous outcomes such as hospital length of stay.
We will use generalized linear mixed effects models to account for multiple admissions and clustering of patients within hospitals. Hospitals will be included as random effects in these models and the clustered nature of the data will be modelled by fitting the appropriate covariance structure. Analysis will be performed using the PROC GLIMMIX procedure available in SAS software.
The project has been approved by the Monash Health Human Research Ethics Committee (RES-17-0000-238L), and site-specific approval has been obtained at each site prior to commencement of data collection.
The rate of administration of IV therapy for children in Australia and New Zealand is currently unknown. Based on the approximately 3% rate of IV therapy in the UK study,11 we anticipate the need to review approximately 1,000 eligible charts at each site to obtain 30-40 presentations where IV bronchodilators are administered. Data collection at nineteen hospitals will provide data on approximately 570 children with asthma requiring IV therapy. Each site will be requested to collect data on 1,000 patients or 12 months of data (whichever is less).
If more than 1,000 eligible medical records are identified on the initial screen of medical records, a random number generator will be used to create a series of random numbers corresponding to the medical records. Screening will then occur in order of the generated random numbers, until 1,000 eligible records are identified.
This project will provide information on incidence of severe paediatric asthma / wheeze in Australia and New Zealand; prevalence of complications and clinical outcomes; and patterns of IV and other escalated therapy for these children. This information will inform discusions relating to the feasibility of specific clinical outcome measures to power trials in children with severe asthma exacerbations. For example, if a particular clinical outcome (e.g. intubation, pneumothorax) was found to be very rare, it would not be feasible to power a randomized controlled study to detect a clinically meaningful difference between treatment groups for that outcome.
Step 5: Consensus meeting and next steps.
Once results from steps 2, 3 and 4 are available, a half day face-to-face meeting will be arranged in Melbourne, Australia, with the ability for international investigators to participate by teleconference. Invitations will be sent to all study steering group members, Delphi study participants who have indicated a willingness to be involved in such a meeting, and representatives of major stakeholders such as general practitioners, patient advocacy groups and specialist medical and nursing colleges.
The meeting will enable discussion of the results of the Delphi survey to agree a final set of core outcomes, and undertake additional voting if required. Attendees will be sent a reminder of their personal Delphi scoring prior to the meeting. We will seek to ratify outcomes which have met the predetermined definition for the Delphi survey, and to further discuss and then make a decision upon those outcomes which had not met the consensus threshold prior to the meeting. Additional discussion will then occur on how best to measure the chosen outcomes, and identification of any further research required to better inform decisions regarding measurement.
Once the meeting has been completed, we intend to produce a consensus statement relating to core outcome measures in acute severe asthma. This will be drafted and a copy circulated to all those involved in the project for comment prior to finalization and distribution.
At the time of submission of this paper (August 2019), the trial status is as follows:
- Systematic review of outcome measures: completed and published in November 2018.
- Overview of Cochrane Reviews: Protocol completed and published March 2018. Literature search and data collection complete. Manuscript preparation ongoing.
- Patient interviews (protocol version 1.4, date: 17th September 2018). Data collection commenced February 2019. Anticipated completion mid-2020.
- Clinician interviews (protocol version 1.3, date: 31st August 2018 ). Data collection commenced May 2019. Anticipated completion mid-2020.
- Review of clinical practice guidleines (protocol version 1.3, date: 3rd August 2018). Data collection commenced September 2018, data extraction ongoing. Anticipated completion mid-2020.
- Retrospective cohort study (current protocol version 2.8, date: 12th December 2018 – amended due to additional participating hospitals; original approved protocol version 2.3, date 30th May 2017). Data collection commenced June 2017, data collection ongoing. Anticipated completion late 2020.
- Yet to commence: Delphi study, review of asthma severity scores, consensus meeting.