The protocol for this meta-analysis was registered in the PROSPERO database of prospectively registered systematic reviews (www.crd.york.ac.uk/PROSPERO; CRD42019146859), and the completed study conforms to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [21].
Search Strategy And Selection Criteria
A systematic literature search was performed in eight electronic databases: PubMed, EMBASE, PsycINFO, Scopus, the Cochrane Library, OVID, ERIC, and Web of Science. Each database was initially searched for relevant literature in English from its inception through June 13, 2019. We developed a search strategy for PubMed based on MeSH (Medical Subject Headings) terms and text words from key research that we identified a priori (see Table S1 for the full search strings). We reviewed the reference lists of key publications and relevant narrative reviews to identify studies that might have been missed in the database searches. To check for possible publication bias, we also undertook a gray literature search in clinical trial registries (http://www.ClinicalTrials.gov) using identical inclusion criteria to identify unpublished trials.
After the removal of duplicates, two independent investigators performed title scans and abstract reviews, and they screened the full-text articles to assess their eligibility for inclusion. Concordance among the investigators was satisfactory, with a positive agreement of 0.83; any disagreements between the authors were resolved by consultation with the third investigator. A number of prespecified inclusion and exclusion criteria were used to select key studies. The inclusion criteria were as follows: (a) randomized controlled trials (RCTs), quasi-experimental studies (i.e., nonequivalent control group design, one-group pre-test/post-test design), and natural experiments (a form of observational study in which the researcher cannot control or withhold the allocation of an intervention to particular areas or communities; thus, natural or predetermined variation in allocation occurs); (b) longitudinal studies with at least one assessment in early childhood and one in middle childhood or adolescence; (c) mean age of participants at first (“early child”) assessment < 5 years; (d) mean age of participants at last (“middle child or adolescent”) assessment ≥ 6 years; (e) professional/clinical diagnosis of ASD, autism, PDD-NOS, or Asperger syndrome based on DSM criteria; (f) English language articles published in a peer-reviewed journal (dissertations were excluded); and (g) articles assessing the effectiveness of a CTM and reporting some primary outcome variables focused on child functioning.
The following exclusion criteria were applied: (a) studies including children with medical complications or who were receiving drug treatment.; (b) pharmacological or dietary interventions, focused intervention practices (FIP, e.g., Pre-school Autism Communication Trial (PACT), Joint Attention, Symbolic Play and Engagement Regulation (JASPER)), and other interventions with unclear evidence according to National Institute for Health and Care Excellence (NICE) guidance, such as secretin, chelation, or hyperbaric oxygen therapy; (c) studies reporting on a CTM that was not present in at least two other studies, that is “isolated intervention approaches”; and (d) pre- and posttest means and standard deviations were not available after attempts to contact the authors and could not be calculated from the descriptive data or statistical tests in the study manuscript. For multiple studies on the same cohort, we selected the publication with the longest follow-up, provided it included results with detailed demographic and intervention information.
Data Extraction And Quality Assessment Of The Included Studies
Pairs of investigators independently performed data extraction with a predesigned standardized form, and discrepancies were resolved by repeated discussion until consensus was reached. To ensure the accuracy and completeness of the extracted information, the third investigator repeatedly verified the extracted data abstraction for all the included studies. The following information from each included study was extracted: first author; region, study design, and year of publication; population characteristics at intake, including subtype of sample, age, and sex (% male); intervention characteristics, including intervention approaches (e.g., EIBI, ESDM), setting (clinical/home), delivery agent (therapists/parents), intensity and duration in weeks; type of comparison (e.g., treatment as usual, no comparison group); assessment times (i.e., pre, post, follow-up); the measures employed in each study; and the outcomes reported in middle childhood (e.g., autism symptomatology, IQ, adaptive behavior, language).
Two independent investigators applied the Evaluative Method for Determining Evidence-Based Practices in Autism to assess the quality of the included studies [22], which is available for many study designs. A previous study suggested that this tool can be applied to evaluate intervention studies and produce valid assessments of the empirical evidence on practices in children with ASD [23]. Six primary and eight secondary quality indicators were applied and are annotated in Table S3, including the characteristics of the participants, independent variables, dependent variables, comparison conditions, random assignment, blinding of raters, and fidelity. Divergence between the two investigators who evaluated the quality of the studies was resolved by discussion. The quality of a study was assessed as “strong” when all the primary indicators received high quality ratings and there were four or more secondary indicators; “adequate” when more than four primary indicators received high ratings, with no unacceptable ratings and evidence of at least two secondary indicators; and “weak” otherwise.
Calculation Of Effect Sizes (ESS)
Because the instruments for evaluating a given outcome differed across studies (e.g., Wechsler Intelligence Scale for Children vs. Merrill-Palmer Scales of Mental Tests), we used standardized ESs to obtain standardized measurements of the effect of the intervention on the outcome variables. According to the methodology of Reichow and Wolery [24], two types of ES were computed: the standardized mean change ES (gc) and the standardized mean difference (SMD) ES (gd). We took two steps to ensure the most conservative ES. First, ESs were calculated only when the data necessary for the calculation were available. If an outcome variable was missing the necessary data for the calculation of an ES, no ES was calculated for that outcome of the study. Hence, no data were extrapolated or interpolated for the calculation of ESs. Second, ESs based on small samples are known to be biased [25], so we multiplied them by the small sample correction factor [26].
The first ES analyses were calculated for the intervention groups in all the included studies and examined the differences between the average gains made by distinct samples. This comparison revealed the absolute difference within a sample from pre-intervention to middle childhood without regard to the comparison group in between-group studies. We calculated the gc by dividing each adjusted mean change by the pooled standard deviation.
For the between-group studies, the gd was used to show the magnitude of the difference between the group receiving a CTM and the comparison group. The ES (gd) was calculated by dividing each adjusted mean difference by the pooled standard deviation.
Meta-analytic Procedures
We combined findings from all the included studies using prespecified meta-analytic methods to determine the effect of CTMs in middle childhood in children with ASD. Two data synthesis steps: (1) Meta-analysis I was performed to estimate longitudinal changes in broader outcomes in middle childhood in children with ASD who were exposed to a CTM. (2) Meta-analysis II was performed to assess the effect of CTMs on those outcomes in the test group compared to the control group. The standardized mean change/difference and 95% confidence interval (CI) for each intervention effect were the primary outcome measures in the meta-analysis. Due to the diversity in population characteristics and intervention approaches, we expected a conservative estimation of the ESs. Consequently, a meta-analysis was performed on studies judged sufficiently similar and appropriate to pool using random effects models. Cohen’s criteria [27] were applied to determine the magnitude of the effect. The magnitude of the effect was assessed as “trivial” when the ES was < 0.2, “small” when the ES was between 0.2 and 0.49, “medium” when the ES was between 0.5 and 0.79, and “large” when the ES was ≥ 0.8.
Prespecified and exploratory stratified analyses were conducted to assess differences in ESs based on the use of (1) EIBI, (2) ESDM, and (3) other interventions to examine the consistency of the intervention approaches. Outcomes reported in fewer than six studies and parental outcomes were discarded from the meta-analysis, and studies were rank-ordered by quality rating in the forest plots.
The I2 statistic was used to assess the potential heterogeneity of ESs across interventions. An I2 > 50% was considered evidence of heterogeneity. Sensitivity analysis was performed by reanalyzing the data using a fixed effects model and by omitting one study at a time to assess the impact of each individual study on the overall pooled estimate. Potential publication bias was assessed in two ways: a funnel plot and Egger’s linear regression test. When publication bias was identified, a nonparametric trim-and-fill method was used to adjust for the publication bias.
Meta-regression
Across 9 predictors in univariate meta-regressions (Table 3), three mediators of longitudinal change in middle childhood outcomes emerged: (1) EIBI was more effective in reducing symptom severity than non-EIBI programs, and this explained 64% of the heterogeneity (Coefficient=-1.31, P = 0.045). (2) Higher total and social adaptive functioning were associated with longer total hours of the intervention and explained 78% and 100% of the heterogeneity (Coefficient = 0.0001, P = 0.021; Coefficient = 0.0002, P = 0.032, respectively). (3) Higher social adaptive functioning was also associated with a higher risk of bias (Adj R2 = 100.00%, Coefficient = 0.78, P = 0.026). No potential confounding factors affected the change in DLS. Regarding the multivariate meta-regressions, they demonstrated a clear effect of delivery agent (therapist or therapist and parents) on IQ after the p-value was adjusted (P = 0.028, Table 4). Specifically, the involvement of parents in implementing intervention strategies had a more beneficial effect on IQ enhancement than the involvement of a therapist alone.
Table 3
Results of the univariate meta-regression analyses by adaptation and symptomatic variables.
| | ASD SS | | Composited | | DLS | | Social |
| | Coeff | P | | Coeff | P | | Coeff | P | | Coeff | P |
Internal Validity |
Risk of bias | | 1.100 | 0.33 | | 0.450 | 0.16 | | 0.019 | 0.97 | | 0.780 | 0.03* |
Sample size | | 0.020 | 0.41 | | -0.014 | 0.15 | | -0.037 | 0.63 | | -0.033 | 0.62 |
Population Characteristics |
Pre age | | -0.080 | 0.17 | | -0.018 | 0.48 | | -0.046 | 0.46 | | -0.027 | 0.61 |
Pre IQ | | -0.029 | 0.75 | | 0.001 | 0.98 | | -0.014 | 0.74 | | 0.011 | 0.83 |
Time intervala | | -0.002 | 0.85 | | -0.002 | 0.65 | | 0.009 | 0.53 | | -0.022 | 0.03 |
Post ageb | | -0.005 | 0.64 | | -0.001 | 0.89 | | 0.0006 | 0.98 | | -0.039 | 0.05 |
Intervention Characteristics |
Approachesc | | -1.310 | 0.05* | | -0.704 | 0.18 | | -0.550 | 0.30 | | 0.330 | 0.47 |
Total treatment hours | | -0.0002 | 0.19 | | 0.0001 | 0.02* | | -0.0001 | 0.82 | | 0.0002 | 0.03* |
Delivery agents | | 1.180 | 0.15 | | 0.097 | 0.77 | | 0.120 | 0.84 | | 0.033 | 0.95 |
Notes: a Time interval between postintervention and follow-up. |
b Mean age of participants at last (“middle child or adolescent”) assessment. |
c Categorical variable, EIBI = 1, non-EIBI (ESDM and other interventions) = 0. |
d Based on the result of sensitivity analysis, Magiati (2011) was removed in the meta-regression analyses. |
ASD SS, ASD symptom severity; Coeff, unstandardized meta-regression coefficient; Composite, Vineland adaptive composite score; DLS, Daily living skills; Pre, preintervention. |
* p ≤ 0.05 |
ASD symptom severity - Approaches: Adj R2 = 64.19% |
Vineland adaptive composite score - Total treatment hours: Adj R2 = 78.06% |
Vineland social adaptive score - Total treatment hours: Adj R2 = 100.00% |
Vineland social adaptive score - risk of bias: Adj R2 = 100.00% |
Table 4
Results of the multivariate meta-regression analyses by cognitive function.
| Coefficient | SE | 95% CI | P | tau2 | k | Adj R2 (%) | Model P | Type I errorsa |
IQ | | | | | | | | | |
Delivery agentsb | 0.6756 | 0.2637 | [0.0881, 1.2632] | 0.028* | | | | | |
Pre age | -0.0289 | 0.0204 | [-0.0742, 0.0165] | 0.187 | 0.1294 | 14 | 52.15 | 0.048* | not |
Total treatment hours | 0.00000184 | 0.000046 | [-0.0001, 0.0001] | 0.969 | | | | | |
Notes: a Monte Carlo permutation test was applied to correct type I errors for multiple covariate meta-regressions. |
b Categorical variable: therapist = 1, therapist + parents = 2. |
CI, confidence interval; Coefficient, unstandardized meta-regression coefficient; CTM, comprehensive treatment model; IQ, intelligence quotient; k, number of studies or "clusters"; Pre, preintervention; SE, standard error. |
* p ≤ 0.05 |