The PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) statement, which includes a 17-item checklist, was used to facilitate and structure this systematic review protocol.
Study eligibility criteria
Types of studies
Randomized controlled trials (RCTs) will be included in this systematic review, which include cluster RCTs, wait-list controlled RCTs and cross-over RCTs. Quasi-experimental trials will be excluded.
Children and adults who are prescribed NCS treatment, either as monotherapy or in combination with other treatments, for AR, associated allergic rhinoconjunctivitis (ARC) or rhinosinusitis (RS) are included. We will include individuals with condition subtypes including both seasonal/perennial AR and ARC and acute/chronic RS. Studies that additionally target parents or carers of participants (e.g., children) who take part in the management of NCS treatment adherence are also included. Individuals exclusively on other treatments not including NCS (e.g., antihistamines or immunotherapy treatment) are outside the scope of this systematic review and will be excluded. Interventions which exclusively target health care professionals will be excluded.
Studies will be included if they deliver interventions with primary or secondary aims of improving adherence to NCS through the use of mHealth devices. The WHO definition of mHealth will be used for this systematic review which is a “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices”(19, p. 6). Therefore, studies that implement mHealth devices such as mobile phones, smartphones, smartwatches, tablets, PDAs and electronic monitoring devices (EMDs) as an integral part of the intervention will be included. Such devices do not include touchscreen tablets, telephones, computers or other devices that are not handheld and mobile. Peripheral devices (e.g., sensors, sensory wearables) and web-based programmes will be included as long as they are accompanied by one or more of the above-mentioned primary devices.
These mHealth devices can make up the entire intervention or be part of a broader multifaceted intervention which can be with or without health care professional-to-patient contact (i.e., face-to-face or virtual consultations).
Lastly, studies that exclusively use phone calls or tele-consultations as an alternative to face-to-face consultations will be excluded from this systematic review.
We will only include studies with comparator groups made up of participants who are not provided or do not have access to an mHealth intervention for improving NCS adherence. Comparator groups can either receive usual care or the same intervention devoid of the mHealth intervention component. Usual care will pertain to standard care per guidelines or standard care in the given setting and time in which the study was conducted. We will include studies with multiple intervention arms such as varying types of mHealth interventions as long as one comparator group matches the above description.
The following outcomes are divided into primary and secondary outcomes. The reporting of one or more of the following primary outcomes constitutes an inclusion criterion for this systematic review. However, all reported study outcomes will be extracted unless deemed trivial.
- Symptoms as measured by a subjective assessment.
- Quality of life (QoL) assessed by a validated subjective assessment.
- Adherence to nasal corticosteroids assessed by objective and/or validated subjective assessments.
- Usage of mHealth intervention, using quantitative measures of usage.
- Acceptability of the mHealth intervention, using a validated quantitative instrument such as questionnaires. Qualitative acceptability assessments will be excluded from this systematic review.
- Nasal patency as measured by an objective test.
- Adverse effects.
As some outcomes may be listed as composite measures, all relevant composite and individual outcomes will be extracted as reported in the included studies. If outcomes are presented for numerous time-points, these will all be extracted. No restrictions related to length of follow-up will be applied.
Report eligibility criteria
No restrictions will be put on geographical location or type of setting. Furthermore, we will include studies written in English and in other languages if these are easily translatable using Google Translate to the degree that study characteristics are clearly discernible post-translation.
Only studies that are available in full-text will be included. Attempts will be made to contact appropriate authors to obtain full-text articles if these are not readily-available through the institutional holdings available to the authors of this systematic review. Supplementary reports or conference abstracts that do not include full-text reports will be excluded. Lastly, due to the fast-paced nature of mHealth research, only studies from 2010-present will be eligible for inclusion.
Searches for relevant studies will be conducted in MEDLINE (OVID interface), EMBASE (OVID interface) and CENTRAL (Cochrane Central Register of Controlled Trials) (Wiley interface) and are planned to commence in August 2020. The snowball technique will be used to scan reference lists for additional eligible publications. Furthermore, the bibliography of the included studies will be circulated to all the systematic review authors as well as key experts identified by the authors.
ClinicalTrials.gov, the UK Clinical Research Network Study Portfolio, the Meta Register of Controlled Trials and the first 100 hits on Google Scholar will be searched for relevant unpublished or in-progress trials.
The search strategy will be formed using the “pearl-growing” method in MEDLINE (OVID interface), in which relevant Medical Subject Headings (MeSH), their entry terms and their “term-tree” will be explored, as well as input from the team of authors. The search terms will be validated by a medical librarian with expertise in systematic review searching. A draft of the MEDLINE search strategy is presented in supplemental table 1. The MEDLINE search strategy will be adapted and translated to the other electronic bibliography databases as to adhere appropriately in syntax and MeSH terms. No search limits or filters will be added to individual searches, apart from publishing year range (January 2010 – August 2020).
The database search results will be imported into the reference management program Mendeley 1.19.3 (Mendeley Ltd.) in order to compile the results and remove any duplicates that might be present, through using the programme’s main functions and through manual identification. Once appropriate duplicates have been removed, the remaining citations will be uploaded to Covidence, a web-based systematic review data management programme, which facilitates collaboration between reviewers during the study screening process and maintains the integrity of the systematic process by mirroring the PRISMA statement workflow. A screening form will be developed by the authors for title/abstract and full-text screening based on the eligibility criteria, of which a draft is presented in supplemental table 2. Prior to formal screening commencement, the screening form will be piloted and calibrated accordingly.
Two review authors (MSB, HT) will be blinded to the other’s verdicts and independently conduct the two-stage screening of titles and abstracts of study reports extracted from the search results, respectively, using the developed screening form. Initially, studies that clearly do not meet the inclusion criteria based on their titles will be excluded, while abstracts will be scoured against the inclusion criteria in the abstract phase. During this phase, all reports that meet the inclusion criteria will be coded as “Yes” and otherwise “No” in Covidence, while reasons for exclusion will be noted. Where doubt regarding eligibility occurs, these will be marked as “Maybe” and will be included in the full-text screening for further scrutiny. Subsequently, all full-text study reports will be retrieved for the studies bearing “Yes” or “Maybe” labels to be screened by the two review authors (MSB, HT). Additional information will be sought from the respective authors if necessary, to resolve disagreements regarding eligibility or address uncertainties regarding incomplete or ambiguous methods that require further clarification. Disagreements will be resolved either through discussion or by a third review author (JS), while reasons for exclusion will be documented. All studies identified through the snowball technique will be subjected to the same processes as highlighted above. All of the review authors will not be blinded to either study authors, journal titles or institutions. Cohen’s kappa will be presented to help assess the inter-rater agreement between the two main reviewers (MSB, HT). Multiple reports of the same study will be collated, so that each study, opposed to each report, is the principal unit of interest. The selection process will be documented through a PRISMA flow diagram .
Data collection process
The data extraction will be performed by the aid of a data extraction form which will be developed and modified in Covidence. The template for intervention description and replication (TIDieR) was used to model the data extraction form. Two review authors (MSB, HT) will independently extract data from each included study. Both review authors (MSB, HT) will participate in calibration exercises prior to data extraction commencement. Again, disagreements will be discussed and, if necessary, be resolved by a third review author (JS).
Once developed, the data extraction form will be piloted on at least one of the included studies, while appropriate changes will be made iteratively. Extracted data will be divided into the following six distinct groupings: general study information, methodology, demographic information, intervention details and study outcomes. A draft of the data extraction form can be seen in supplemental table 3.
The following data items will be extracted from the included studies:
- General study information: author(s), institution(s), sponsorship source(s), conflicts of interest, country, setting.
- Methods: study design, date of study, methods of randomisation, length of follow-up, total study duration, length of “run-in” periods, study centre details, recruitment setting(s), recruitment methods.
- Participants: N participants (baseline and upon completion), gender, median age, range of age, sub-population groups, condition type(s), condition classification(s), co-morbidities, inclusion criteria, exclusion criteria, comparison between groups at baseline and mHealth device familiarity.
- Interventions: intervention aim(s) (primary and secondary aims), intervention details, type of intervention(s) (theory or non-theory-based), intervention administrator(s), type of mHealth device(s), mHealth device name(s) (e.g., app names), device make and model (if issued), non-mHealth intervention component(s), description of mHealth training (if administered), intervention modification(s), intervention retention and mHealth adherence/usage rates.
- Comparison: comparison group descriptions.
- Outcomes: Details about primary and secondary outcomes, including their individual value(s), data type(s), type of effect measure(s), assessment method(s) and reported time-points.
Upon completion of data extraction, data will be transferred to the review manager RevMan 5 by review author (MSB), and subsequently cross-checked in the study reports by review author (HT).
Outcomes and prioritisation
- Symptoms will be a primary outcome as measured by a patient-reported outcome (PRO) assessment, which is based on scores related to nasal symptoms (sneezing, rhinorrhoea, nasal pruritus and nasal congestion) and ocular symptoms (itchy eyes and watery eyes). Such assessments can include:
- Symptom scores, for example RTSS (Rhinoconjunctivitis Total Symptom Score), CQ5 (Congestion Quantifier Five-item test) or similar.
- Disease control scores, for example VAS (Visual Analogue Scales), CARAT (Control of Allergic Rhinitis and Allergy Test) or similar.
- Combined symptom-medication scores (CSMS), for example the Allergy-Control-SCORE (ACS) or similar.
- The second primary outcome will be disease-specific QoL (Quality of Life) as measured by a validated PRO assessment such as the RQLQ (Rhinoconjunctivitis Quality of Life Questionnaire), PRQLQ (Paediatric Rhinoconjunctivitis Quality of Life Questionnaire) or similar.
- Adherence to NCS assessed by direct/indirect objective or validated PRO assessments can include the following:
- Direct/indirect objective assessments can include directly observed therapy, NCS cannister weighing, refill record-based adherence measurement such as MPR (Medication Possession Ratio), PDC (Proportion of Days Covered) or similar.
- Validated PRO assessments can include the MMAS-8 (Morisky Medical Adherence Scale), MARS (Medication Adherence Report Scale) or similar instrument.
- The usage of the mHealth intervention itself can include assessing system usage data in which it is quantitatively captured how each participant uses the mHealth device. Depending on the mHealth device, this can be measured through number of short message service responses sent, application logins/sessions, modules completed, features used, pages viewed or similar measurement.
- The acceptability of the mHealth intervention can be measured through a validated instrument such as SUTAQ (Service User Technology Acceptability Questionnaire) or similar measurement.
- Nasal patency which can be assessed by objective measurements such as PNEF (Peak Nasal Expiratory Flow), PNIF (Peak Nasal Inspiratory Flow), NAR (Nasal Airway Resistance) by anterior or posterior rhinomanometry or similar assessment.
- Adverse events.
Reported outcomes will be subdivided into the three follow-up categories:
- Short-term, between 1-11 weeks.
- Medium-term, 12-26 weeks
- Long-term, ≥27 weeks.
If there are multiple reported outcomes that fall within each follow-up category, the primary study time-point and longest follow-up time will be given priority in that order.
Risk of bias individual studies
A risk-of-bias assessment will be made for each individual study outcome which has been defined as a primary outcome (symptoms, QoL and NCS adherence) for this systematic review. The risk-of-bias assessment will be carried out independently by two review authors (MSB, HT) using the Cochrane risk of bias tool 2.0 using Excel version 16.37 (Microsoft Corporation) as platform. We aim to investigate the effect of assignment to intervention, or the “intention to treat”, and will use the risk-of-bias tool accordingly. Prior to assessment, efforts will be made to contact study authors if study protocols and trial registry records are not available to the review authors. Using the tool, the assessment of risk of bias will be conducted using the following domains (as outlined in table 8.2a in the Cochrane Handbook for Systematic Reviews of Interventions):
- Bias arising in the randomisation process.
- Bias due to deviations from intended interventions.
- Bias due to missing outcome data.
- Bias in measurement of the outcome.
- Bias in selection of the reported outcome.
- Overall bias.
For each domain, a series of “signalling questions” pertaining to the assessment of risk of bias will be answered with either “yes”, “probably yes”, “probably no”, “no” and “no information”. An algorithm will map the answers recorded and propose a risk-of-bias judgement of either “low risk of bias”, “some concerns” or “high risk of bias” for each domain, which can be overridden by the review authors if deemed appropriate. Comments and direct quotations from study reports and protocols will be attached to support answers given to each signalling question. Likewise, justification will be provided if risk-of-bias judgements from the algorithm are overridden. Lastly, the domain-level judgements will provide the basis for an overall risk-of-bias judgement for each specific outcome being assessed for each study. Domain-level and overall risk-of-bias consensus judgments will be presented in an appropriate table format. Review authors will not be blinded to study details.
Any disagreements will be resolved through discussion or by involving a third review author (AS) to arbitrate if necessary.
Criteria for quantitative synthesis
If included studies have sufficient clinical and methodological homogeneity, meta-analyses will be conducted preferring to use a random-effects model over fixed-effect model due to the anticipated between-study demographic- and intervention-specific diversity. More specifically, outcomes, outcome measures, follow-up categories and comparator groups will be compared to assess homogeneity and meta-analysis feasibility. A chi-square test will be provided but will not determine meta-analysis feasibility.
Measures of treatment effect
In the presence of adequate homogeneity, outcome data will be processed and analysed using RevMan 5. For dichotomous data, risk ratio (RR) with a 95% confidence interval (CI) will be preferred over odds ratio (OR) as a summary statistic, if possible. Appropriate data will be extracted from included studies to calculate RR, if RRs are not reported as effect measures.
For continuous data, mean differences with 95% CIs between intervention groups will be used where measurement scales are identical. Standardised mean differences with 95% CIs will be used if differing measurement scales are found. All calculations will be made in accordance with the statistical guidelines as referenced in the Cochrane Handbook for Systematic Reviews of Interventions.
Unit of analysis issues:
For cluster RCTs, the intra-cluster correlation coefficient will be extracted to modify the outcome estimates in the absence of analysis appropriately accounting for the effect of clustering, as recommended in the Cochrane Handbook for Systematic Reviews of Interventions. For multi-arm intervention group RCTs, ineligible treatment-arm effects will be discounted while eligible treatment arms and comparison groups will be combined to create a single pair-wise comparison as outlined in the Cochrane Handbook for Systematic Reviews of Interventions.
For cross-over RCTs and wait-list controlled RCTs, only data from the first period prior to cross-over or control group receiving the mHealth intervention will be used to mitigate biased estimates due to the anticipated lack of blinding.
Handling missing data:
Original study authors will be contacted in efforts to obtain missing data or study characteristics relevant for data synthesis. If adequate numerical outcome data is not obtained, the study will be excluded from the meta-analysis.
Subgroup analysis and assessment of heterogeneity
In the presence of adequate statistical power, sub-group analyses will be conducted sub-dividing participants according to age group (children & adolescents: 5-17 years vs. adults: ≥18 years) and mHealth feature (unidirectional vs. bidirectional participant engagement) as these are predicted to be the most influential a priori effect modifiers. The random-effects model will be used. A meta-regression will be performed if the number of included studies exceed 10.
We intend to carry out sensitivity analyses, in which the following trials are removed from the primary outcome analysis:
- Trials labelled with a “high” overall risk of bias for the individual main outcome based on the risk-of-bias assessment.
- Trials using unvalidated PRO assessment methods (e.g., symptom-medication scores).
A systematic narrative synthesis will be provided irrespective of meta-analysis feasibility. Study characteristics, clinical outcomes (i.e., symptoms, quality of life, adherence to NCS, nasal patency, adverse events) and process outcomes (i.e., mHealth usage, intervention acceptability) will be summarised and explained via appropriate text and tables. A “characteristics of included studies” table will be presented and ordered by mHealth feature (unidirectional and bidirectional participant engagement). The intra- and inter-relationships and findings of included studies will be explored in accordance with the guidance for narrative synthesis from the Centre for Reviews and Dissemination.
Outcome reporting bias will be assessed through comparing outcomes reported in a study protocol with those in the published report. Where a study protocol is unavailable, outcomes listed in the methods and results sections of the published report will be compared to each other to assess discrepancies.
In the event that more than 10 studies are pooled, a funnel plot using Egger’s regression test will be created and assessed for publication biases.
Confidence in cumulative estimate
In case of meta-analysis feasibility, a “summary of findings” table will be produced in RevMan 5 to include the primary outcomes (symptoms, QoL and NCS adherence) as defined for this systematic review. The quality of evidence for the primary outcomes will be assessed across five domains (risk of bias, consistency of effect, imprecision, indirectness and publication bias). Additional domains will be added if deemed appropriate. Ultimately, the quality of evidence for each outcome will be graded as either “high” (there is a low likelihood that further research will produce a substantially different effect), “moderate” (there is a moderate likelihood that further research will produce a substantially different effect), “low” (there is a high likelihood that further research will produce a substantially different effect), or “very low” (there is a very high likelihood that further research will produce a substantially different effect). The methods and recommendations described in chapter 14 of the Cochrane Handbook for Systematic Reviews of Interventions and in the Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach will be used, using the GRADEpro software.