A team of investigators with expertise in critical care EEG, neurocritical care, adult and pediatric neurology, and health information collaborated to develop the research question and study design. This systematic review will follow the guidelines set out in the Cochrane Handbook for Systematic Review and Meta-Analyses. The protocol has been registered in PROSPERO and is undergoing review by the editorial team. The research methodology presented in the final manuscript will adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Deviations from the protocol will be recorded in the final report.
Information sources and search strategy
MEDLINE, EMBASE, Web of Science (WoS), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane Central Register of Controlled Trials (CENTRAL) will be systematically searched from their inception date to January 2020. With the collaboration of a health information specialist, the search strategy was carefully developed to capture all studies of potential interest, using a combination of free text keywords and subject headings terms. Keywords used to construct the search included “electroencephalography,” “critical care,” “emergency care,” “curriculum,” and “medical education.” An example of our full search strategy for MEDLINE has been provided as a supplementary file with this protocol.
A review of the gray literature will be performed to identify unpublished or ongoing studies using Google Scholar, https://clinicaltrials.gov, and http://www.controlled-trials.com. This systematic review will also include a hand search of the past 10 years of published abstracts from relevant conference proceedings. Before submission for publication, the search will be rerun through each database to account for newly reported findings. Authors of included studies will be contacted up to 3 times to clarify any unclear or unavailable information as necessary. A step-by-step breakdown of included and excluded studies will be provided in the final manuscript using the standard PRIMSA flow diagram.
Eligibility criteria and study selection
We will apply the following eligibility criteria to identify studies for inclusion in this review: (1) randomized controlled trials, pre-post interventional studies, and observational studies including descriptive and cohort studies (2) examining adult or pediatric patients in whom EEG recording (including intermittent/short EEG, processed EEG (e.g., Bis, entropy, Sedline), video EEG, cEEG, and QEEG) was performed. (3) Implementation of EEG must occur in either the intensive care unit (ICU), emergency department (ED), or post-anesthetic care unit (PACU) setting, (4) with data analysis performed by a non-expert in EEG interpretation (e.g. bedside nurse, intensivist, ED physician, anesthetist, or trainee physician). (5) Studies must include a program of structured training in EEG interpretation. Cluster RCTs will not be included, since the unit of allocation of included studies must be the individual. Studies will be excluded if they (1) do not adequately describe the structure and content of training programs, (2) are applied to subjects with specialized knowledge of EEG (e.g. neurophysiologists, epileptologists), (3) or if they are conducted in a setting other than the ICU, ED, or PACU. Studies involving medical students, residents, and clinical fellows will be eligible for consideration as long as they meet the remainder of the inclusion criteria. Records will be screened and managed by two reviewers independently using Endnote, version 9.0 (Clarivate Analytics).
Two reviewers (ST, WA) will independently screen all material generated by the search algorithm to identify studies for inclusion. Each reviewer will be blinded to the others’ appraisal of the literature. Studies will initially be screened by title, keywords, and abstract to determine article eligibility. Articles passing the initial step will be reviewed by two reviewers in full, with articles selected based on eligibility criteria. As a further step, the reference list of each selected study will be scanned to identify additional studies for inclusion. In cases of ambiguity, authors of studies in question will be contacted for clarification of uncertain information, with their response rates tracked and reported in the final manuscript. No restrictions will be applied to language and foreign papers will be translated to English. Abstracts without a corresponding full-length manuscript will be considered if they present sufficient material on educational design, implementation, and outcomes. In cases of incomplete information, authors of abstracts will be contacted up to 3 times to obtain further details of the study; studies will be excluded in the event of non-response. In the event of discrepancies in the final study list generated by the two reviewers, a third reviewer will be consulted for arbitration. Reasons for exclusion of studies assessed in full will be presented in the final published manuscript.
Two independent reviewers (ST, WA) will extract data from each study in the final list into a standardized pre-piloted data collection form. We will collect and report data on: (1) study design, including but not limited to study type, year of publication, inclusion and exclusion criteria, treatment setting (e.g., ICU, ER, or PACU), sources of funding, and conflicts of interest; (2) baseline patient characteristics including age, sex, comorbidities, and primary pathology (neurological (e.g., traumatic brain injury, subarachnoid hemorrhage, epilepsy) vs non-neurological (e.g., sepsis, acute respiratory distress syndrome (ARDS), trauma)); (3) indication for EEG; (4) type of medical or surgical treatment received, including use of mechanical ventilation or tracheostomy; (5) characteristics of the EEG educational program, including mode of instruction (e.g., self-guided, didactic, case-based); (6) duration of the training program, and whether follow-up sessions were organized; (7) methods for assessing trainee performance (e.g. written quiz, bedside interpretation of EEG recordings); (8) outcome of the intervention (e.g. improvement in EEG interpretation capabilities by the trainee); (9) and trainee feedback on the educational intervention (if reported). The initial data extraction form will be piloted on five included studies to ensure robustness, with subsequent modifications for thoroughness performed if necessary. A template of the data extraction form is appended in the supplementary file. Discrepancies in extracted data will be resolved in discussion between the two primary reviewers. Duplicated studies will be included only once in the final analysis, with the most comprehensive article being represented.
Assessment of methodological quality and risk of bias
If our search identifies a randomized controlled trial (RCT) deemed eligible for inclusion, its risk of bias will be evaluated by two independent reviewers with the Cochrane Collaboration’s risk of bias tool. However, we anticipate the majority of studies to be descriptive studies and pre-post interventional studies. For the assessment of these studies, we will use the National Institute of Health (NIH) quality assessment tools. NIH checklists will be adapted for each study design. Each checklist includes items for evaluating potential flaws in study methods or implementation, including sources of bias (e.g., patient selection, performance, attrition, and detection), confounding, study power, the strength of causality in the association between interventions and outcomes, and other factors. Quality reviewers may select "yes," "no," or "cannot determine/not reported/not applicable" in response to each item on the tool. Studies are assigned a quality rating of “good”, “fair”, or “poor” based on the aggregate total of “yes” responses. Summary reports on the quality of each represented study will be presented in tables in the final manuscript. The assessment of both randomized and non-randomized studies will be undertaken independently by two reviewers (ST, WA), with discrepancies resolved after joint article review and discussion.
Our primary objective is to comprehensively review educational programs for training non-experts to interpret EEG, with a focus on reporting programs’ structure, patterns of implementation, challenges, and trainee experience. Results will be presented as a narrative summary, with information grouped by theme (e.g. program structure, methods of evaluation, trainee experience). As secondary objectives, we will i) aim to compare the performance of non-experts receiving structured training in EEG interpretation against a gold standard (e.g. EEG interpretation by an epileptologist or neurophysiologist), and ii) compare trainees’ performance pre- and post- completion of the educational program. At the minimum, the ability of trainees to accurately detect seizures will be compared with experienced interpreters. Additionally, if studies permit, trainees will also be assessed in their ability to recognize other important EEG abnormalities, including (but not limited to) alpha coma, periodic lateral epileptiform discharges (PLEDs) and generalized periodic discharges (GPEDs). If available, performance in the assessment of raw EEG and QEEG by non-expert personnel will be also assessed.
Agreement in interpretation of EEG between non-neurologists and experts will be assessed with the kappa coefficient (k), and the strength of agreement will be interpreted with the Landis and Koch classification (<0 = very bad; 0 – 0.2 = slight; 0.21 – 0.4 = fair; 0.41 – 0.6 = moderate; 0.61 – 0.8 = good; and 0.81 – 1 = almost perfect). For the assessment of both secondary objectives, only studies reporting the relevant data will be included.
Statistical analysis and data synthesis
Data will be extracted from the standardized form and presented in a descriptive manner. Categorical data will be reported in proportions while continuous data will be presented as means with standard deviations or medians with ranges depending on the format used in the primary studies. We expect considerable heterogeneity in educational design, training duration, and use of performance metrics, potentially prohibiting meta-analysis. The Cochrane Handbook outlines methods to synthesize findings if meta-analysis cannot be performed, for example, due to heterogeneity. In our review, we will follow the Cochrane methods if meta-analysis is deemed inappropriate. In addition, we will adhere to a rigorous reporting methodology as described by the synthesis without meta-analysis (SWiM) guidelines. The SWiM guideline is a 9-item reporting checklist that outlines standardized metrics used for synthesis, the synthesis method, a summary of findings, and limitations of the synthesis. A protocol for the synthesis and analysis of studies is presented using the SWiM format, although modifications may be required based on the final pool of available studies.
- Grouping studies for synthesis: Studies being analyzed quantitatively will be separated by design into two groups. Pre-post interventional studies assessing non-expert performance before and after an EEG training program will constitute the first group. Studies assessing non-expert performance against a gold standard (e.g., EEG interpretation by an electroencephalographer) will constitute the second group. For individual studies reporting both measures, data and outcomes will be reported separately.
- Standardized metric used: For the comparison of EEG interpretation by non-experts with experts, the kappa coefficient will be used to standardize assessments. If the kappa statistic is not reported, it will be calculated using data from the corresponding 2x2 table. In cases of incomplete data, authors will be contacted to supply the data. For pre-post comparisons, the chosen measure of intervention effect will be the P-value. If the P-value is not presented, data from the study will be used to perform the P-value calculation. In cases of incomplete data, authors of the study will be contacted.
- Synthesis method: For non-expert vs. expert comparisons, a forest plot of the summary statistic (i.e., kappa coefficient) will be displayed along with the corresponding 95% confidence intervals for each study. Studies will be pooled using random effects models (DerSimonian and Laird method), and a summary estimate will be calculated as a weighted average of the kappa values from individual studies. Statistical calculations will be performed using Review Manager 5.3. For pre-post studies, we will use random effects models to calculate pooled estimates of effect sizes using Review Manager 5.3.5 software (Cochrane Collaboration, Oxford, UK). Pooled continuous-effect measures will be expressed as mean differences (MD) with 95 % confidence intervals (CI). Assuming there are a reasonable number of studies, we will plan to perform a subgroup analysis to examine for heterogeneity of the effect across subgroups. The following subgroups will be examined: clinical background of the participant (e.g., nurse or physician), duration of the training program, and type of EEG reviewed (e.g., aEEG, CDSA, short/intermittent EEG). We will perform a z test of interaction for all subgroup comparisons, which tests the null hypothesis that the intervention effects in each subgroup are the same. If there is considerable clinical heterogeneity, we will alternatively adhere to the SWiM guided synthesis by combining the P-values based on the direction of effect and the precise P-value of individual studies using Fisher’s method. Results will be displayed visually in the form of an albatross plot, following the recommendation from the Cochrane Handbook .
- Criteria used to prioritize results: No restrictions will be applied to studies being synthesized. However, given that studies are likely to vary in their risk of bias, the final quality of each study (“good”, “fair”, or “poor”) as determined by the NIH quality assessment tool will be displayed alongside the results to contextualize readers’ interpretation of the synthesis.
- Investigation of heterogeneity: If formal statistical analysis is possible, we will use the I2 statistic to evaluate percentage variance that is attributable to study heterogeneity. Interpretations regarding the significance of heterogeneity will be made as per standard characterization as negligible (<40%), moderate (30-60%), substantial (50-90%), or considerable (75-100%). In case formal statistical analysis is not possible, informal methods will be used to investigate heterogeneity in reported effects. Tables and figures will be ordered in the final manuscript based on hypothesized modifiers, including methodologic characteristics (e.g., study design), subpopulations (e.g., clinical background of the participants, number of years in practice, setting where the program was administered (e.g., ICU or ED)), and other contextual factors. Where possible, results from studies will be grouped and presented according to degree of heterogeneity. This method is in line with the Cochrane Handbook’s recommendation to group studies by characteristics that might enhance their interpretation.
- Certainty of evidence: Assessment of the certainty of evidence will take place according to the GRADE recommendations. However, assessing some domains is anticipated to be challenging, and if formal assessment cannot be adequately performed, this step will be excluded in the final analysis. For the assessment of publication bias, funnel plots will be constructed and visually inspected for asymmetry.
- Data presentation methods: Data will be presented in figures and tables in the final manuscript, grouped by the secondary outcome of interest. Forest plots of kappa coefficients will be constructed for studies comparing seizure detection between non-expert vs. expert, as described above. For pre-post studies, data will either be presented as forest plots of the mean test scores or p-values will be displayed as an albatross plot.
- Reporting results: For each secondary objective, the synthesized findings will be described, making clear which studies contributed to the findings and the synthesis method used. Findings will not be over-interpreted and will be presented in the context of the quality of studies informing them.
- Limitations of the synthesis: Limitations will be described, including those resulting from limited evidence, incompletely reported outcome, or incomplete effect estimates.