Two types of data will be extracted from the studies: general information about the included studies and specific information about the methods used to evaluate interventions.
Data concerning the characteristics of the studies
These information will help to describe the included studies and the contexts in which the interventions are implemented. These information concern the authors, the title of the article, the date of publication, the place where the studies were carried out, the purpose and the results of the studies.
Data on impact evaluation methods
A data extraction grid was developed for the purposes of this review (Table 2). This grid has three sections that allow for the extraction of: i) information about the design of the studies, ii) information about the statistical methods (when appropriate), iii) information about the mathematical methods (when appropriate). The items used in the extraction grid are based on tools and recommendations consistent with each of the above-mentioned sections and with the investigated research question in the review [10–12]. We also verified that the information to be extracted are consistent with the standards of reporting according to the study design; the CONSORT statement and the relevant extensions for randomized controlled studies [13–16], the TREND statement for non-randomized controlled studies [17] and the STROBE statement for observational studies [18].
To test the applicability of the grid, a sample of 14 articles selected on the basis of their content was used (Additional file 2): 6 articles concerning randomized controlled studies (2 of which present mathematical impact evaluation), 3 articles concerning nonrandomized controlled studies, and 5 articles concerning observational studies (2 of which present mathematical impact evaluation). These articles were selected independently of the research questions addressed in the review to be conducted. The item development for each section of the extraction grid is presented below.
i) Study design information
Eligible studies are based on strictly quantitative designs, i.e. randomized controlled studies, non-randomized controlled studies or observational studies. Items to extract study design-specific information originate from the Mixed Methods Appraisal Tool (MMAT). This tool allows to assess the design quality of quantitative studies [19, 20, 11]. Because these questions are specific to study design, the information to be extracted are different according to the study type. These items offer a choice of three possible answers (yes, no, unclear).
ii) Information about statistical methods
Items concerning statistical methods are formulated using the guidelines for reporting Statistical Analyses and Methods in the Published Literature (SAMPL) [10]. The criteria identified in these guidelines are related to the implementation of statistical methods. The developed items allow the extraction of information common to different statistical methods for purpose to compare included studies. These items can be categorized into two types: i) those that answer questions about the sample size, ii) those that answer questions about the implementation of statistical methods in the analyses. The formulated items offer a choice of four possible answers (yes, no, imprecise, not concerned) except for one item allowing the classification of statistical methods (seven categories: correlation analysis, regression analysis, ANOVA/ANCOVA, bayesian analyses, statistical tests, other, not concerned).
iii) Information about mathematical methods
Items concerning mathematical methods are formulated using the guidelines for Strengthening The Reporting of Empirical Simulation Studies (STRESS) [12]. Several criteria identified from these guidelines allow the collection of information which are consistent with several types of models. The developed items are based on these criteria and can be categorized into four types: i) description of the objectives of the model, ii) description of the assumptions made by the model, iii) description of the data used in the model, and iv) description of the implementation of the model. These items offer a choice of four possible answers (yes, no, unclear, not concerned).
Table 2
Section
|
Items
|
Study design informations
|
Randomized controlled trials
|
Is randomization appropriately performed?
|
Are the groups comparable at baseline?
|
Are there complete outcome data?
|
Are outcome assessors blinded to the intervention provided?
|
Did the participants adhere to the assigned intervention?
|
Non-randomized controlled trials/Observational studies
|
Are the participants representative of the target population?
|
Are measurements appropriate regarding both the outcome and intervention (or exposure)?
|
Are there complete outcome data?
|
Are the confounders accounted for in the design and analysis?
|
During the study period, is the intervention administered (or exposure occurred) as intended?
|
Information about statistical methods
|
Is the minimum difference considered important to prove the intervention efficacy/effectiveness reported?
|
Are the alpha and beta levels that define statistical significance reported and respected?
|
What is the statistical method used to assess the intervention impact on the primary outcome?
|
What is statistical the method used to assess the intervention impact on HIV prevalence/incidence?
|
Did the authors report analyses to confirm that the statistical assumptions of the analysis were met or the goodness of fit of the model?
|
Did the authors report a measure of precision alongside the impact measure?
|
Is the design taken into account in the statistical analyses?
|
Information about the mathematical methods
|
Are the parameters to be estimated in the model specified?
|
Are the uncertainties concerning these parameters evaluated?
|
Are the different components of the model and their roles clearly defined?
|
Are the different scenarios/processes/assumptions tested by the model clearly specified?
|
Are the data and data sources specified?
|
Are the states of the various model components clearly specified from data?
|
Are the methods used to compute estimates specified?
|