The protocol is reported in accordance with the reporting guideline provided in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) statement (see Additional file 1) (51, 52), and is registered in the International Prospective Register of Systematic Reviews (PROSPERO CRD42020196492) database. The review will be carried out following recommendations outlined in The Cochrane Handbook of Systematic Review of Interventions (47), and PRISMA (53-55).
Criteria for considering studies for this review
Types of studies
We will search for and include randomized clinical trials, irrespective of publication status, publication year, and language. We will also include quasi-randomized studies and observational studies identified during our search for trials for the assessment of harms, but we will not conduct searches for these studies. The reason for including such observational studies is that randomized clinical trials often do not report rare adverse events or late occurring adverse events.
Types of participants
Any participant not previously infected with SARS-CoV-2. Participants will be included irrespective of sex, comorbidities, immune status, and risk group.
Types of interventions
We will include any vaccine aiming to prevent COVID-19, i.e. all vaccines listed in Table 1 or any other vaccine irrespective of dose and duration of administration. We will group any vaccine with the same mechanism (e.g. RNA, DNA, viral vector, protein subunit, etc.) and target (e.g. S-protein). Authors blinded to the data extraction of results will group the vaccines into groups with similar vaccines. The authors involved in this process will be blinded for authors of the trials and will not have access to the values of outcome data at this point.
We will include randomized clinical trials with any control group, i.e. head-to-head comparisons versus placebo, ‘active placebo’ (a matching placebo that produces noticeable adverse effects that may convince the participant being vaccinated), usual care (or similar terms), no intervention, another vaccine aiming at preventing COVID-19, or any other ‘active’ comparator. We will accept any of these control interventions irrespective of dose and duration of administration.
Co-interventions will be allowed provided they are administered equally to the comparison groups.
- All-cause mortality.
- Proportion of participants with confirmed COVID-19 (verified by RT-PCR or similar laboratory tests).
- Proportion of participants with one or more serious adverse events. We will use the International Conference on Harmonization of technical requirements for registration of pharmaceuticals for human use—Good Clinical Practice (ICH-GCP) definition of a serious adverse event, which is any untoward medical occurrence that resulted in death, was life-threatening, required hospitalization or prolonging of existing hospitalization, and resulted in persistent or significant disability or jeopardized the participant (56). If the trialists do not use the ICH-GCP definition, we will include the data if the trialists use the term ‘serious adverse event’. If the trialists do not use the ICH-GCP definition nor use the term serious adverse event, then we will also include the data if the event clearly fulfills the ICH-GCP definition for a serious adverse event. We will exploratorily assess each type of serious adverse event separately (see below).
- Health-related quality of life (assessed on any valid continuous scale).
- Proportion of participants with one or more adverse events not considered serious. We will exploratorily assess each type of adverse events not considered serious separately (see below).
- SARS-CoV-2 neutralising antibody titers.
- SARS-CoV-2 IgG-binding antibody titers.
- Proportion of participants with seroconversion for SARS-CoV-2 neutralising antibody (defined as either a 4-fold increase from baseline or higher than 2 SDs above mean in the control group).
- Proportion of participants with seroconversion for SARS-CoV-2 IgG-binding antibody (defined as either a 4-fold increase from baseline or higher than 2 SDs above mean in the control group).
- Individual types of serious adverse events will be analyzed separately.
- Individual types of adverse events not considered serious will be analyzed separately.
We will use the trial results reported at maximum follow-up for all outcomes. We will also assess the time points 14 days and 28 days for the exploratory serological outcomes.
Search methods for identification of studies
An experienced information specialist will search Cochrane Central Register of Controlled Trials (CENTRAL), Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica database (EMBASE), Latin American and Caribbean Health Sciences Literature (LILACS), Science Citation Index Expanded (SCI-EXPANDED),
Conference Proceedings Citation Index—Science (CPCI-S), Chinese Biomedical Literature Database (CBM), China Network Knowledge Information (CNKI), Chinese Science Journal Database (VIP), and Wafang Database to identify relevant trials. We will
search all databases from their inception to the present. Trials will be included irrespective of language, publication status, publication year, and publication type. For a detailed search strategy for all electronic searches, see Additional file 2.
Searching other resources
We will identify additional references by manually searching the references of articles from the computerized databases. We will also search special COVID-19 trial sites, including a website with living mapping and living systematic review of COVID-19 studies (https://covid-nma.com/), a website developed by Vaccine Centre at the London School of Hygiene & Tropical Medicine (https://vac-lshtm.shinyapps.io/ncov_vaccine_landscape/), The Lancet’s “A real-time dashboard of clinical trials for COVID-19 (https://www.thelancet.com/journals/landig/article/PIIS2589-7500(20)30086-8/fulltext), the preprint server for health sciences, www.medrxiv.org, and an open, accessible and frequently updated clinical trial registration for COVID-19 trials (https://doi.org/10.12688/wellcomeopenres.15821.1)
We will also search online trial registries such as ClinicalTrials.gov (clinicaltrials.gov), the Chinese Clinical Trial Registry (www.chictr.org.cn), the European Medicines Agency (EMA) (www.ema.europa.eu/), the World Health Organization (WHO) International Clinical Trials Registry Platform (www.who.int/ictrp/), and the Food and Drug Administration (FDA) (www.fda.gov/) for ongoing or unpublished trials. We will contact experts in the field and pharmaceutical companies to enquire about additional trials. We will search for grey literature in the System for Information on Grey Literature in Europe OpenGrey (www.opengrey.eu).
Data extraction and management
Two review authors will independently extract data from included trials in a predefined form. Disagreements will be resolved by discussion, or if required, through consultation with a third author (JCJ or CG). The two review authors will assess duplicate publications and companion papers of a trial together to evaluate all available data simultaneously (maximize data extraction, correct bias assessment). Each trial will be named after the first author and year of the primary publication, and all secondary publications will be classified under that name. We will contact the trial authors by email to specify any missing data, which may not be reported sufficiently or not at all in the publication.
We will search for information regarding industry funding of either personal or academic activities for each trial author. We will note in the “Characteristics of included studies” table if outcome data were not reported in a usable way. Two review authors will independently transfer data into the Stata file (57).
Living systematic review
A living systematic review is defined as a systematic review which is continually updated and incorporates relevant new evidence as it becomes available (58). This methodology may be particularly important in the COVID-19 pandemic, where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions (58).
There are four fundamental differences between conventional systematic reviews and living systematic reviews: publication format, work processes, author team management, and statistical methods (59). In this living systematic review, two independent investigators will receive an updated literature search file and include relevant newly published or unpublished trials once a week. The relevant meta-analyses, Trial Sequential Analyses, and network meta-analyses will continuously be updated, and if new evidence is available (judged by the steering committee of the LIVING VACCINE review), the results will be published. Every month, the steering committee will discuss whether searching once a week is necessary. The living systematic review process will be initiated September 21, 2020. For an illustration of the living systematic review workflow, see Fig. 1 (with permission from Juul et al.(19) and Systematics Reviews).
We will extract the following data: bias risk components (as defined below), trial design (parallel, factorial, crossover, cluster), estimation of sample size, inclusion and exclusion criteria, number of intervention groups, and length of follow-up.
We will extract the following data: number of randomized participants, number of participants with comorbidities and types of comorbidities, number of analyzed participants, number of participants lost to follow-up/ withdrawals/crossover, age range (mean or median), and sex ratio.
Experimental intervention characteristics
We will extract the following data: type of vaccine, type of adjuvants, dose of intervention, duration of intervention.
Control intervention characteristics
We will extract the following data: type of control intervention, dose of intervention, duration of intervention.
All outcomes listed above will be extracted from each randomized clinical trial. For each outcome, we will identify if outcomes are missing, inappropriately measured, or selectively reported according to the criteria described later in the ‘missing outcome data’ bias domain, the ‘risk of bias in measurement of the outcome’ bias domain, and the ‘risk of bias in selection of the reported result’ bias domain.
Assessment of risk of bias in the included studies
Our bias risk assessment will be based on the Cochrane Risk of Bias tool—version 2 (RoB 2) as recommended in The Cochrane Handbook of Systematic Reviews of Interventions (47). We will evaluate the methodology in respect of the following bias domains.
Bias arising from the randomization process
Low risk of bias Allocation was adequately concealed, AND there are no baseline imbalances across intervention groups at baseline appear to be compatible with chance, AND an adequate (random or otherwise unpredictable) method was used to generate allocation sequence, OR there is no information about the method used to generate the allocation sequence
Some concerns Allocation was adequately concealed, AND there is a problem with the method of sequence generation, OR baseline imbalances suggest a problem with the randomization process, OR no information is provided about concealment of allocation, AND baseline imbalances across intervention groups appear to be compatible with chance, OR no information to answer any of the signaling questions.
High risk of bias Allocation sequence was not concealed, OR no information is provided about concealment of allocation sequence, AND baseline imbalances suggest a problem with the randomization process.
Bias due to deviation from intended interventions
Low risk of bias Participants, carers, and personnel were unaware of intervention groups during the trial, OR participants, carers, or personnel were aware of intervention groups during the trial but any deviations from intended intervention reflected usual practice, OR participants, carers, or personnel were aware of intervention groups during the trial but any deviations from intended intervention were unlikely to impact on the outcome, AND no participants were analyzed in the wrong intervention groups (that is, on the basis of intervention actually received rather than of randomized allocation).
Some concerns Participants, carers, or personnel were aware of intervention groups and there is no information on whether there were deviations from usual practice that were likely to impact on the outcome and were imbalanced between intervention groups, OR some participants were analyzed in the wrong intervention groups (on the basis of intervention actually received rather than of randomized allocation) but there was little potential for a substantial impact on the estimated effect of intervention.
High risk of bias Participants, carers, or personnel were aware of intervention groups, and there were deviations from intended interventions that were unbalanced between the intervention groups and likely to have affected the outcome, OR some participants were analyzed in the wrong intervention groups (on the basis of intervention actually received rather than of randomized allocation), and there was potential for a substantial impact on the estimated effect of intervention.
Bias due to missing outcome data
Low risk of bias No missing data OR non-differential missing data (similar proportion of and similar reasons for missing data in compared groups) OR evidence of robustness of effect estimate to missing data (based on adequate statistical methods for handling missing data and sensitivity analysis).
Some concerns An unclear degree of missing data or unclear information on proportion and reasons for missingness in compared groups AND there is no evidence that the effect estimate is robust to missing data.
High risk of bias A high degree of missing data AND differential missing data (different proportion of or different reasons for missing data in compared groups) AND there is no evidence that the effect estimate is robust to missing data.
Bias in measurement of outcomes
Low risk of bias The outcome assessors were unaware of the intervention received by study participants, OR the outcome assessors were aware of the intervention received by study participants, but the assessment of the outcome was unlikely to be influenced by knowledge of the intervention received.
Some concerns There is no information available to determine whether the assessment of the outcome is likely to be influenced by knowledge of the intervention received.
High risk of bias The assessment of the outcome was likely to be influenced by knowledge of the intervention received by study participants.
Bias arising from selective reporting of results
Low risk of bias Reported outcome data are unlikely to have been selected, on the basis of the results, from multiple outcome measurements (e.g. scales, definitions, time points) within the outcome domain, and reported outcome data are unlikely to have been selected, on the basis of the results, from multiple analyses of the data.
Some concerns There is insufficient information available to exclude the possibility that reported outcome data were selected, on the basis of the results, from multiple outcome measurements (e.g. scales, definitions, time points) within the outcome domain, or from multiple analyses of the data. Given that analysis intentions are often unavailable or not reported with sufficient detail, we anticipate that this will be the default judgment for most trials.
High risk of bias Reported outcome data are likely to have been selected, on the basis of the results, from multiple outcome measurements (e.g. scales, definitions, time points) within the outcome domain, or from multiple analyses of the data (or both).
Overall assessment of risk of bias
Low risk of bias The trial is judged to be at low risk of bias for all domains for this result.
High risk of bias The trial is judged to be at high risk of bias or to be at some concerns in at least one domain for this result. Our subgroup analysis will compare the intervention effect of trials at low risk of bias to trials at high risk of bias, that is one or more domains at some concern or high risk of bias.
We will assess the domains ‘missing outcome data’, ’risk of bias in measurement of the outcome’, and ‘risk of bias in selection of the reported result’ for each outcome result. Thus, we can assess the bias risk for each outcome assessed in addition to each trial. Our primary conclusions will be based on the results of our primary outcome results with overall low risk of bias. Both our primary and secondary conclusions will be presented in the Summary of Findings-tables.
We will assess confidence in network meta-analysis results using CINeMA (Confidence in Net-work Meta-Analysis) (60-62).
Differences between the protocol and the review
We will conduct the review according to this published protocol and report any deviations from it in the ’Differences between the protocol and the review’ section of the systematic review.
Measurement of treatment effect
Dichotomous outcomes We will calculate risk ratios (RRs) with 95% confidence interval (CI) for dichotomous outcomes, as well as the trial sequential analysis adjusted CIs (see below). Peto’s odds ratio (OR) with 95% CI will be used where the number of observed events is small (less than 5% of sample per group), and treatment groups are balanced (63).
Continuous outcomes We will calculate the mean differences (MDs) or in case of different measurement scales the standardized mean difference (SMD) with 95% CI for continuous outcomes. We will analyze change from baseline scores using a MD if the same scale is used across studies. For different measurement scales in the same analysis model, we will use the SMD effect size. In case some studies do not report change scores, but provide follow-up values, we will combine them together in a single model using MD (47). We will also calculate trial sequential analysis-adjusted CIs (see below).
Dealing with missing data
We will use intention-to-treat data if provided by the trialists (64). We will, as the first option, contact all trial authors to obtain any relevant missing data (i.e. for data extraction and for assessment of risk of bias, as specified above), when individual patient data is not available.
Dichotomous outcomes We will not impute missing values for any outcomes in our primary analysis. In our sensitivity analyses (see sensitivity analysis), we will impute data.
If standard deviations (SDs) are not reported, we will calculate SDs using relevant trial data (e.g. P values), if available. We will prefer intention-to-treat data, but if the original report did not contain such data, per protocol data will be used. In our best-worst worst-best scenarios (see sensitivity analysis) for continuous outcomes, we will impute data.
Assessment of heterogeneity
We will primarily investigate forest plots to visually assess heterogeneity. We will secondly quantify heterogeneity using the I2 statistic (47, 65, 66) and will estimate the between-study variance using the restricted maximum likelihood method (67, 68). We will investigate evident heterogeneity through subgroup analyses (see “Subgroup analyses and integration of heterogeneity” section below). We may ultimately decide that a meta-analysis should be avoided if heterogeneity is high (47). To assess the magnitude of heterogeneity we will compare the estimated amount with the distribution by Rhodes et al for continuous and Turner et al for dichotomous data (69, 70).
Assessment of reporting biases
We will use a funnel plot to assess reporting bias if ten or more trials are included (47). We will visually inspect funnel plots to assess for small-study effects, accounting for its potential limitations (e.g. low power) (47). From this information, we will assess possible reporting bias. For dichotomous outcomes, we will test asymmetry with the Harbord’s test (71) if τ2 is less than 0.1 and with the Rücker test if τ2 is more than 0.1 (47). For continuous outcomes, we will use the regression asymmetry test (72) and the adjusted rank correlation (73).
Unit of analysis issues
We will include randomized clinical trials for assessment of benefits and harms.
In case of trials with a cross‐over design, we will include the data from the first trial period in order to avoid residual effects from the treatment (47). In order to avoid repeated observations on trial participants, we will use participant trial data at the longest follow‐up (47).
We will analyze cluster randomized trials using the procedures referenced in the Cochrane Handbook for Systematic Reviews of Interventions (47). Where results did not control for clustering, we will contact trial authors to request an estimate of the intracluster correlation coefficient (ICC). If the trial authors are unable to provide an ICC, we will calculate the ICC using design effects (74).
If we during our searches for trials identify observational studies reporting on harms, we will tabulate these harms and report them separately in the Results section. This is to cover rare and late occurring harms. We will tabulate the types of adverse events (serious and non‐serious) that are reported in the non‐randomized studies retrieved with the searches for randomized clinical trials. This will limit the information on harms in our systematic review. If benefits of certain vaccines are found, then systematic reviews of harms, based on observational studies, should be conducted (75).
Aggregate data meta-analysis
We will undertake the aggregate meta-analyses according to The Cochrane Handbook of Systematic Reviews of Interventions (47), Keus et al. (76), and our eight-step assessment suggested by Jakobsen et al. (48). We will use the statistical software Stata version 16.1 (command: meta) to analyze data (57). We will assess our intervention effects with both a random-effects meta-analysis (DerSimonian and Laird method) (46) and fixed-effect meta-analysis (Mantel-Haenszel method) for each treatment comparison separately (77). We will report the more conservative point estimate of the two (48). The more conservative point estimate is the estimate with the highest P value or the widest confidence interval. If there is substantial deviation between the random-effects and fixed-effect meta-analyses, we will report and discuss the results. We will assess a total of three primary outcomes and two secondary outcomes, and we will therefore consider a P value of 0.0167 or less as the threshold for statistical significance (48). We will investigate heterogeneity through subgroup analyses. We will use the eight-step procedure to assess if the thresholds for significance are crossed (48). Where multiple trial arms are reported in a single trial, we will include only the relevant arms. If two comparisons are combined in the same meta-analysis, we will halve the control group to avoid double-counting (47). Trials with a factorial design will be included. In case of, e.g. a 2×2 factorial designed trial, the two groups receiving COVID-19 vaccination will be considered experimental groups, while the two groups receiving a placebo, ‘active placebo’, standard care, no intervention, or ‘active’ comparator will be considered control groups.
Trial Sequential Analysis
Due to the continuous inclusion of new trials and hence repetitive testing of accumulating data when updating reviews, there is an increased risk of type I error. We wish to control the risks of both type I errors and type II errors. We will therefore perform Trial Sequential Analysis on all outcomes, in order to calculate the diversity-adjusted required information size (DARIS; that is, the number of participants needed in a meta-analysis to detect or reject a certain intervention effect) and the cumulative Z-curve’s breach of relevant trial sequential monitoring boundaries (49, 50, 78-84). A more detailed description of Trial Sequential Analysis can be found in the manual (78) and at http://www.ctu.dk/tsa/. For dichotomous outcomes, we will estimate the required information size based on the observed proportion of patients with an outcome in the control group (the cumulative proportion of patients with an event in the control groups relative to all patients in
the control groups). When assessing the reduction of confirmed COVID-19 we will conduct three trial sequential analyses with a relative risk reduction of 20%, 50%, and 70% respectively. The 50% relative risk reduction will be our primary analysis. We will use a relative risk reduction of 20% for the remaining dichotomous outcomes, an alpha of 1.67% for all our outcomes, a beta of 10%, and the observed diversity as suggested by the trials in the meta-analysis. For continuous outcomes, we will in the Trial Sequential Analysis use the observed standard deviation (SD), a mean difference equal to the observed SD/2, an alpha of 1.67% for all outcomes, a beta of 10%, and the observed diversity as suggested by the trials in the meta-analysis.
We will obtain information about the interventions of interest either from head-to-head trials, or from trials comparing a COVID-19 intervention with placebo, standard care, no intervention, or “active placebo.” Hence, the synthesis comparator set consists of all the interventions listed in the background section as well as a placebo, ‘active placebo’, standard care, no intervention, or ‘active’ comparator trials. Each specific vaccine will be analyzed separately and will also be clustered with similar vaccines. We will describe the characteristics of the eligible randomized clinical trials and their populations using frequencies and percentages for categorical data and means and standard deviations for continuous data.
Descriptive statistics will be also generated for each treatment comparison describing important clinical and methodological characteristics (e.g. publication year, participant age). Each outcome dataset will be presented in a different network diagram, where the size of the nodes will be proportional to the total number of randomized participants, and the width of each edge will be weighted according to the number of studies comparing the connected treatments. We will additionally plot the edges of each network according to the average risk of bias per treatment comparison, using green for low, yellow for moderate, and red for high risk of bias. We anticipate that any participant who meets inclusion criteria is, in principle, equally likely to be randomized to any of the interventions in the synthesis comparator set. Network meta-analysis will be performed using Stata 16.1 (command: mvmeta) under the frequentist framework (57) using the network suite of commands (85). The network meta-analysis synthesizes evidence for the comparative effectiveness of more than two alternative interventions for the same condition (86). In case we encounter trials with more than two arms included in our review, we will only include the study once in the table showing the ‘characteristics of included studies’. The latter will also prevent the problem of the trial appearing more than once in the risk of bias assessment and this way also ensures no double counting the number of randomized clinical trials.
We will only perform network meta-analysis if a connected network of trials can be conducted (87).
If network meta-analysis is possible, we will, assess a priori the two prerequisite assumptions: transitivity and consistency. We will assess for the transitivity assumption across treatment comparisons in the network using boxplots, and will evaluate the assumption of consistency using the design-by-treatment interaction model as a global test (66, 86). Effect modifiers will be age, sex, ethnicity/origin, exposure to COVID-19, whether or not immunocompromised/ deficient, whether or not with chronic lung disease. The transitivity assumption for carrying out an NMA will be evaluated using these effect modifiers. We will also explore these through network subgroup meta-analyses (see section below). If we conclude that the transitivity and consistency assumptions are not met, we will not perform network meta-analysis, but will present direct and indirect evidence separately.
The estimation of each treatment comparison will be reported separately using the relevant effect size (RR), a 95% CI, and a 95% prediction interval. We will use the network forest plot to illustrate the summary effect size of the comparative effectiveness among interventions. Along the estimated effect sizes, we will present the ranking probabilities for each treatment being at each possible rank, as well as the surface under the cumulative ranking curve (SUCRA) (88, 89). A rank-heat plot will be used to depict the SUCRA values (and their 95% CI) across all outcomes (90).
We will conduct a random-effects network meta-analysis, assuming a common within network heterogeneity for each analysis, since the nature of the interventions in the network is similar (85, 87).
With earlier iteration of the living review, it is possible that some of the networks may be very sparse, in which case between-study heterogeneity variances may be overestimated leading to wider credible intervals from network estimates (70, 91). If wide CIs are due to rare events as well, we will use the Mantel Haenszel in NMA using R (92).
To make sure that this complex network meta-analysis will be meaningful, relevant, and manageable, we will use the following process to define the nodes of the network. We will independently and in duplicate extract all data from all trials. We will present all the extracted data minus the primary and secondary outcomes for another group of the authors, who based on lists of all the different vaccines (different types, adjuvants, doses, durations) and all the different comparators (different types, doses, durations) will determine the groupings of the meaningful nodes to be compared within each connected network. The authors involved in this process will be blinded for authors of the trials and will not have access to the values of outcome data at this point.
Individual patient data meta-analysis
Results of individual patient data meta-analysis will increase the possibility to identify subgroups of patients with specific effects of the assessed interventions (93-95). It will enable us to calculate the treatment by covariate interactions using patient-level covariates (such as sex and age).
If we receive individual patient data for all eligible randomized clinical trials, we will analyze the data using a one-stage analysis model based on generalized linear mixed models. This analysis will be adjusted for the categoric baseline variables that the trials used as stratifications variables in their randomization (only the common variables that all of the trials adjust for). When analyzing continuous data, we will also adjust all analyses for the baseline value.
If we are unable to obtain sufficient individual patient data, we will secondly conduct a two-stage analysis, where at 1st stage, we will reduce available individual patient data to aggregate data for each study, and at 2nd stage, we will combine all available data in a meta-analysis.
Assessments of underlying statistical assumptions
We will systematically assess underlying statistical assumptions for all statistical analyses (57, 96, 97). In short, for all regression analyses, we will test for major interactions between each covariate and the intervention variable. We will, in turn, include each possible first order interaction between included covariates and the intervention variable. For each combination, we will test if the interaction term is significant and assess the effect size. We will only consider that there is evidence of an interaction if the interaction is statistically significant after Bonferroni adjusted thresholds (0.05 divided by number of possible interactions) and if the interaction shows a clinically significant effect. If it is concluded that the interaction is significant, we will consider both presenting an analysis separately for each (e.g. for each site if there is significant interaction between the trial intervention and ‘site’) and an overall analysis including the interaction term in the model (57, 96, 97). For detailed description of the planned assessments for underlying assumptions, please consult the recommendations of Nørskov et al. (57, 96, 97).
We will perform the following subgroup analyses when analyzing the primary outcomes (all-cause mortality, confirmed COVID-19, and serious adverse events).
- Trials at high risk of bias compared to trials at low risk of bias. This is due to literature demonstrating overestimation of benefits and underestimation of harms in trials at risk of bias (98-100).
- Trials without for-profit bias compared to trials at unknown or known risk of for-profit bias. This is due to literature demonstrating overestimation of benefits and underestimation of harms in trials at risk of for-profit bias (37).
- Type of vaccine (including but not limited to messenger RNA; DNA; non-replicating viral vector; replicating viral vector; inactivated virus; protein subunit; dendritic cell; other vaccines like live-attenuated; polysaccharide vaccine; conjugate vaccines).
- Age (children and adolescents as defined by trialists; adults as defined by trialists; elderly as defined by trialists).
- Type of antibody target (e.g. nonstructural replicase polyprotein, E protein, M protein, N protein, S protein, or other targets).
- Ethnicity (e.g. Asian, Caucasian, Arab, Black, Mixed, other) or participants origin (South East Asian, European, Eastern Mediterranean, African, Western Pacific).
- Sex (male, female).
- Trials including participants who are immunocompromised compared to trials with participant that are not.
- Trials including participants with history of chronic lung disease compared to trials without history of chronic lung disease.
- Trials including unexposed participants at the time of vaccination compared to trials including participants exposed to SARS-CoV-2.
- Trials including vaccines that target a T cell mediated response compared to trials that does not.
- Trials at high or uncertain risks of vested interests compared to trials at low risk of vested interests. This subgroup will assess for-profit bias (37).
- Trials using aluminium adjuvants compared to trials without aluminium adjuvants (101, 102).
- Trials randomizing clusters compared to trials randomizing individual participants.
- Trials using cross-over design compared to trials without cross-over design.
We will use the formal test for subgroup differences in STATA 16.1 (command: meta) (57). We will perform any unanticipated subgroup analyses, if we identify these, as more information about this virus and its treatment becomes available. We will use ICEMAN to assess the credibility of the subgroups (103).
To assess the potential impact of the missing data for dichotomous outcomes, we will perform the two following sensitivity analyses on all primary and secondary outcomes.
We will consider using multiple imputation techniques as recommended by Jakobsen et al. (64). Please consult this publication for a detailed description of the handling of missing data. We will present best-worst and worst-best case scenarios if it is not valid to ignore missing data (48). Best-worst and worst-best case scenarios assess the potential range of impact of the missing data for the trial results.
In the ‘best-worst’ case scenario, it is assumed that all patients lost to follow-up in the intervention group have had a beneficial outcome, and all those with missing outcomes in the control group have had a harmful outcome (48). Conversely, in the ‘worst-best’ case scenario, it is assumed that all patients who were lost to follow-up in the experimental group have had a harmful outcome, and that all those lost to follow-up in the control group have had a beneficial outcome (48). When continuous outcomes are used, a ‘beneficial outcome’ will be defined as the group mean plus two SDs of the group mean, and a ‘harmful outcome’ will be defined as the group mean minus two SDs of the group mean (48).
We will present results of this scenario in our review. Other post hoc sensitivity analyses might be warranted if unexpected clinical or statistical heterogeneity is identified during the analysis of the review or any other unanticipated issues that we learn about COVID-19 along the way that may impact the results (48).
Summary of Findings tables
We will create summary of findings tables including each of the prespecified outcomes (all-cause mortality, confirmed COVID-19, serious adverse events, health-related quality of life, and non-serious adverse events). We will use the five GRADE considerations (bias risk of the trials, consistency of effect, imprecision, indirectness, and publication bias) and CINeMA to assess the quality of a body of evidence (48, 104-106). We will assess imprecision using Trial Sequential Analysis. We will downgrade imprecision in GRADE by two levels if the accrued number of participants is below 50% of the DARIS, and one level if between 50% and 100% of DARIS. We will not downgrade if the cumulative Z-curve crosses the monitoring boundaries for benefit, harm, or futility, or DARIS is reached. We will justify all decisions to downgrade the quality of evidence using footnotes, and we will make comments to aid the reader’s understanding of the review where necessary. Firstly, we will present our results in the Summary of Findings table based on the results from the trials with overall low risk of bias, and secondly, we will present the results based on all trials. We will present the assessment of our three comparisons traditional aggregate data meta-analyses, network meta-analysis, and individual patient data meta-analyses separately. We will discuss all concurring results of these analyses as well as any conflicting results between the three. Two review authors will independently make judgements about the certainty of the evidence, with disagreements resolved by discussion or involving a third review author. We will justify, document, and incorporate judgements into reporting of results for each outcome. We will extract study data, format our comparisons in data tables, and prepare ’Summary of findings’ tables before writing the results and conclusions of our review.
Data sharing and availability
Full syntax of all statistical analyses will be published as supplementary material. All aggregate data will be published regularly. Anonymized individual patient data will also be published if possible (we will discuss this with the trialists).
Findings of this living systematic review will be published in international peer-reviewed scientific journals. Further, a dedicated webpage for the project will be developed, where iterative versions of the living systematic review will be accommodated with visual illustrations.