Adapted and novel interventions are more effective than adopted interventions: a meta-analytic replication of controlled social intervention research across prevention level and setting

doi:10.21203/rs.3.rs-2184012/v1

Download PDF

Research Article

Adapted and novel interventions are more effective than adopted interventions: a meta-analytic replication of controlled social intervention research across prevention level and setting

https://doi.org/10.21203/rs.3.rs-2184012/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background. Several approaches to the implementation of evidence-based interventions (EBIs) are currently in use. EBIs may be adapted or adopted from previous settings or novel interventions may be developed to achieve public health goals. Within the intervention and implementation literature, a major unsolved dilemma relates to the management of adaptation versus fidelity when EBIs are transferred between settings. An initial attempt to empirically understand this dilemma was made in 2015 which explored meta-analytically the origins of interventions and compared their effect sizes. It was found that adapted interventions produced the highest effect sizes followed by novel and adopted interventions. The current study attempts to replicate these findings using a larger data set.

Methods. We used meta-analytic techniques to compare effect sizes across a population of Swedish efficacy and effectiveness studies. Interventions investigated were categorized into adapted, adopted, and novel interventions. Nine subcategories were explored. In addition, we explored differences in effect sizes between settings as well as the impact of study characteristics on effect size.

Results. Of the 523 studies included, 22% described adapted interventions, 33% adopted interventions, and 45% novel interventions. The largest effect size was found for adapted interventions followed by novel and adopted interventions. Standard mean effects were significantly different from zero across categories. Study characteristics did not have a large impact on effect size, but interventions provided in the mental health setting showed the highest standard mean difference, followed by somatic healthcare and social services.

Conclusions. The results reported here are in line with a growing body of evidence suggesting that there is a need to take the fit between the EBI and the context into account when implementing interventions.

fidelity

cultural adaptation

prevention

efficacy

effectiveness

evidence-based

Underlying the idea that EBIs should be disseminated to achieve widespread public health impact is an understanding that the positive effects found in efficacy and effectiveness trials will be maintained when interventions are moved from their original setting to new settings. A consistent challenge to implementation science is related to the management of the adaptation and fidelity dilemma. However, few empirical examples exist on the relative effects of different approaches to intervention delivery.
We replicate a prior meta-analysis and find that adapted interventions, adopted interventions and novel interventions all have standard mean effect sizes significantly different from zero. However, adapted interventions have the largest mean effect followed by novel and adopted interventions.
These findings add to the scholarly discussion regarding the adaptation fidelity dilemma by uncovering differences across settings and types of adaption and novel approaches.

The science of social intervention (1) is concerned with the systematic development and testing of intentional change strategies (2–4) to promote health, prevent ill health or maintain healthy developmental trajectories. The ultimate goal of providing social interventions being to dramatically impact societal health and wellbeing (5–7) by targeting diverse populations across prevention level (i.e., universal, selective, indicated; 6, 8) and setting (e.g., social services, mental health). In order to bring the science of social intervention to scale for widespread public health gain, however, interventions need first be developed, tested for their efficacy and effectiveness, and finally assessed for their readiness for broad dissemination, where a stakeholder (e.g., policy-maker, practitioner, researcher) interested in using an evidence-based intervention (EBI) makes the decision to put it to use in practice (i.e., scale-up; 9, 10). Although there are several, and somewhat varying, guidelines available and in use for evaluating claims of intervention effectiveness (e.g., 9, 11, 12, 13), there is widespread support for the understanding that in order for an intervention to be deemed evidence-based, it must first be assessed for its effectiveness according to an accepted standard (e.g., 5, 9). Bringing the science of social intervention to scale through the dissemination of EBIs has, however, proved challenging (14).

Underlying the idea that EBIs should be disseminated to achieve widespread public health impact is an understanding that the positive effects found in efficacy and effectiveness trials will be maintained when interventions are moved from their original setting to new settings. A consistent challenge to implementation science is related to the management of the adaptation and fidelity dilemma (15, 16). The dilemma refers to the balance between adopting and delivering an intervention with fidelity while also assuring the intervention fits the local context which may necessitate a certain level of adaptation to the intervention under consideration. The current literature acknowledge that adaptations can be needed when using an EBI while it also highlights the importance of implementing interventions with fidelity (17–19).

Implementation fidelity refers to the degree to which a given intervention is implemented as intended by developers and in relation to previous intervention trials when interventions are transferred to new settings (20, 21). A common argument is that positive effects can only be maintained if interventions are implemented with fidelity in their new setting (9, 20). However, documented challenges exist to the maintenance of intervention fidelity when transferring interventions across settings (22) and there is a growing literature supporting the idea that planned adaptation is necessary when transferring interventions across settings to maintain effectiveness (23, 24). For a given stakeholder interested in using a given EBI the dilemma of adaptation versus fidelity needs to be solved: adapt the EBI to the local context or adopt the EBI as described previously with fidelity.

When the focus of implementation of an EBI is to maintain fidelity, otherwise known as adoption, stakeholders are expected to deliver the EBI without diverging from the given instructions on how the intervention is to be delivered. This includes, delivering the intervention in the same way (e.g., 30-minute sessions, 8-week treatment, respectively), using the same format delivery and channel (e.g., group or individual setting; digital or face-to-face), and with the same content as implemented in a prior setting (15). How fidelity looks may then differ depending on if the EBI is manual based, where the delivery guidance is explicit and generally highly specified, or if it is a general practice or a broad standardized approach (e.g., Responsible Beverage Service training), where there may be general guidance or material available but no manual.

When an EBI is to be adapted to a new setting, the adaptations can vary widely across dose, length, content, format, setting and target population (25–27). Target population adaptations can be adaptations to an EBI being implemented to a population for which the EBI was not originally designed (28). For example, Ahmad, Larsson (29) took a manualized eye movement desensitization and reprocessing treatment and adapted the material to suit children and their developmental level. Format adaptations are adaptations made to the delivery of an EBI. For example, Calbring, Gunnarsdóttir (30) took a published self-help book and implemented the material in an online program, thus adapting the delivery of the treatment. Pragmatic adaptations include adaptations to the session or material length. For example, Livheim, Hayes (31) adapted a manualized ACT protocol to be given in six weeks rather than eight weeks, due to time constraints. Cultural adaptations are those where the EBI is adapted to fit the cultural norms of the target setting or population (32–34). For example, Kling, Forster (35) implemented a United States devised parent training program to a Swedish context, however they culturally adapted the program by removing time-outs, as this was not an accepted disciplinary approach in Sweden.

In addition, many social interventions in use today are locally developed (either tested or untested; 36, 37). Thus, stakeholders have a third option available to them when approaching the provision of a social intervention: develop a novel intervention. Ultimately, the decision regarding which strategy to choose will rest on an understanding of which strategy will best serve the population in question. An increased understanding of how these approaches impact client outcomes may help increase our understanding of how to transfer interventions across settings to achieve the more widespread goal of impacting societal health and wellbeing. However, the available evidence supporting the approach to scale-up and spread – adapt an existing EBI to fit the local context, adopt an existing EBI with fidelity or develop a novel intervention – is limited.

To address the knowledge gap concerning the impact of these approaches to intervention delivery, we have previously conducted a meta-analysis of social interventions delivered in Sweden and Germany (36). The findings across samplessuggested that adapted, adopted and novel interventions were all effective. However, results from the Swedish sample showed that adapted interventions were the most effective but made up only a small proportion of the total number of included studies (approx. 10%). In addition, compared to adapted interventions and novel interventions, the group of interventions adopted without any adaptation had the lowest average effect size. This result was stable when study design and sample size were controlled for. However, the Swedish sample in the prior study included only 139 studies and the heterogeneity of interventions (e.g., intervention type, prevention level) limited the ability to draw conclusions about the sub-categories of approaches (e.g., cultural adaptation, pragmatic adaptation). In addition, social interventions are used in different settings, (e.g., social work, mental health, somatic health). The discourse on evidence and what constitutes an evidence-based practice differ between settings as well as the relationship between research and practice, potentially impacting how choice and delivery of interventions are approached (38). Although interventions from different settings and prevention level were included in the prior meta-analysis, the limited sample size did not allow further investigation of these aspects. Thus, the findings need to be replicated and the scope of the questions expanded in a larger sample.

The aim of the current study is to build on the study described above by expanding and focusing on the Swedish findings by including a larger population of studies conducted in Sweden. We (a) further explore the effectiveness of social interventions by replicating an earlier meta-analysis comparing novel interventions, interventions imported from other contexts with adaptation, and interventions adopted from other contexts without adaptation that have been evaluated in controlled research and (b) compare the effectiveness of these different types of interventions in various types of settings. The following questions are asked:

What are the frequencies and characteristics of adapted, adopted, and novel interventions found in the population of efficacy and effectiveness studies published in peer-reviewed journals between 1990–2019?

To what extent are approaches to adaptation, adoption and development of novel interventions related to intervention outcomes?

To what extent are adapted, adopted, and novel interventions implemented in different settings related to intervention outcomes?

Design and setting

We used meta-analytic techniques on the population of efficacy and effectiveness studies conducted in Sweden and published during the investigation period.

Eligibility Criteria

Inclusion criteria

The publication reported on a study which evaluated a behavioral, psychological, or social intervention.
The publication reported on a study which was undertaken in Sweden and the principal investigator was employed by a Swedish university or organization.
The publication was from a scientific journal and was subjected to peer review prior to publication.
The study was published between 1990 and 2019.
The study design reported in the publication was an efficacy, effectiveness, or field experiment using a randomized or non-randomized controlled design.

Exclusion criteria

The study reported did not include an outcome measure at the client, patient, or user level (e.g., only included measures of professional behavior change).
The study reported on an intervention designed to impact somatic health without including at least one behavioral, psychological, or social component.
The study reported on an intervention designed to impact pedagogical or didactical outcomes only (e.g., methods to teach children math skills).

Information Sources

This study is a retrospective analysis and based on information provided in the published literature.

Search Strategy

Although our intent was to conduct a systematic review of published efficacy and effectiveness research conducted in Sweden, a previous study (39) demonstrated that ‘‘Sweden’’ or “Scandinavia” is rarely (approximately 10% of cases) used in the title or among key words in published articles on Swedish intervention research. Therefore, we were unable to use a systematic search strategy (40–42). Despite this we have attempted to follow as closly as possible PRISMA reporting standards (Supplementary information 1) The starting point for the current study was the 139 published articles reporting findings from efficacy and effectiveness research and included in Sundell, Beelmann (36). From this initial pool of studies, the search was carried out in five steps. First, all 191 researchers that were previously identified as having published at least one effectiveness study during the period 1990–2014 (42) were contacted directly. Second, bibliographic searches were conducted to identify research produced by these researchers during the period 2015–2019. Third, the six largest Swedish research funders: the Swedish Research Council for Health, Working Life and Welfare (Forte), the Swedish Research Council for Sustainable Development (Formas), the European Research Council (ERC), the Swedish Research Council (VR), the Swedish Innovation Agency (Vinnova), and the Swedish Crime Victim Authority (BRÅ), database was searched for grants awarded to conduct effectiveness studies during the period 2015–2019 (search terms: evaluation, randomized). Fourth, we conducted a search of planned effectiveness studies registered at clinicaltrials.com (search terms: Swed*, random*, effect*, evaluat*, RCT) as well as studies registered with ISRCTN at www.isrctn.com (search terms: Swed*, mental and behavioral disorder). Fifth, the researchers and studies identified in the search above were then included in searches in unified index EBSCO Discovery (Stockholm University library) (search terms: Swed*, random*, effect*, evaluat*, RCT). Details regarding the search strategy are published previously (41, 42).

The studies identified were then pooled and duplicates removed. If individual study results were reported in several publications, the first publication was used as the source for data extraction and coding.

Data Items And Data Extraction

Study characteristics

Extracted study characteristics include date of publication, study setting (e.g., social services), type of trial (e.g., effectiveness), design (e.g., randomized control), type of control condition (e.g., wait-list control), study population (e.g., adults), follow-up period, prevention level (e.g., indicated), study sample size (excluding dropouts), effect size estimate. If study characteristic (e.g., setting, prevention level) was not explicitly stated in the publication, a determination was made based on the information provided. If we were unable to determine a characteristic by the information provided the data was coded as missing. Data was extracted by two authors with very good (83%) to perfect (100%) agreement on coding of individual items in a random sample of 40 studies (Cohen's kappa 0.63–1.0; 41).

Included intervention

If the study included more than two arms (i.e., experimental, control), the group that was highlighted as the primary focus of the study was selected. If there was no clear information about the priority of the groups, then the first described group was selected and included in this study.

Intervention type

Main intervention type (i.e., adapted, adopted, novel) was further defined in terms of nine mutually exclusive sub-categories (Table 1). Determination of intervention type was based on the information provided in the study. If authors referred to their own previous studies in their intervention descriptions, these studies were referred to for additional information. No additional attempts were made to obtain information outside of that published in the sources described above. The 5 authors independently coded intervention type, by using a coding manual (Supplementary information 2). The coding manual was developed and tested iteratively by the authors. Any disagreements in coding and interpretation of the coding manual were discussed prior to final coding. Interrater reliability on coding of type of intervention was assessed as very good on a random sample of 40 studies (κ = .80 main; κ = .88 subcategories).

Table 1

Broadband-, sub- and definition of categories of intervention as coded in this study
Main category	Subcategory	Definition
Novel	Innovation	The intervention is completely developed based on broad theoretical concepts or models, therapies and/or literature findings.
	Reinvention	The intervention is developed based on empirically established international or national intervention programs, the newly developed program takes inspiration from the referenced programs. The intervention program may also be a re-development of a referenced program due to numerous different modifications.
	Combined	The intervention includes both materials that are newly developed and components from a referenced intervention that are combined into a single treatment. A combined intervention may also include referenced interventions that are given together as a single treatment/program/service.
Adoption	Explicit	The intervention is explicitly adopted as an empirically supported intervention. The intervention is directly translated without any modifications.
	Broadband	The intervention is deemed as a standardized treatment, that may not have an explicit reference but is given as standard (e.g., case management, Responsible Beverage Service training).
Adaptation	Cultural adaptation	The intervention is adapted to suit the cultural norms of the Swedish context.
	Pragmatic adaptation	The intervention is adapted for practical reasons, such as adding or removing materials, increasing, or decreasing number of sessions, or session length.
	Format adaptation	The interventions’ delivery format is adapted (e.g., group therapy given as individual therapy) or a former intervention is adapted to be given online.
	Target population adaptation	The intervention is adapted to suit a specific target population’s needs, such as making material easier to read for children, or modifying language to suit a student population.

Outcome Measures

The primary outcome measure in each study was used and when no such information was provided, the first measure listed in the method section of the publication was used in the analysis.

Statistical Procedures

Effect sizes were calculated according to Lipsey and Wilson (43). When outcomes in a publication were reported to be statistically significant (n < .05) but no additional information on effect size was offered (n = 4), the effect size was set to the average effect for the remaining sample (d = 0.48). When studies stated non-significant effects but without reporting actual effect size (n = 25), the effect size was set to 0. Since extreme effect sizes may have a disproportionate influence on conclusions drawn from statistical analyses, we checked for outliers by using Cook’s distance (44). Three effect sizes were identified as outliers (45). To reduce the impact of these outliers, their values were substituted by one that equaled the highest effect that fell within the normal range (d = 2.3). This had a negligible impact on study results. Comprehensive Meta-Analysis (CMA) software was used to synthesize data. As the sample exhibited significant heterogeneity with a large proportion of random variance, analyses were interpreted from the random effects models (REM). Multiple regression was used to investigate the potential importance of six study characteristics on effect size. The variance inflation factor for the regression analysis varied between 1.04 and 3.20, indicating low severity of multicollinearity (46).

Sensitivity Analysis

As design and sample size have been shown to have significant effects on outcomes, the previous meta-analysis (36) conducted analyses on the entire sample and on a reduced sample in which studies with a quasi-experimental design and studies with sample sizes of less than 50 were removed (Table 3). We replicate these analyses to test the robustness of our results.

In addition, we explore six theoretically important predictors in multiple regression analyses (48) to assess their impact on effect size: RCT or non-RCT, number of subjects, indicted prevention, efficacy trial, active control alternative and number of months in follow-up measurements.

The sample of studies used in the analyses conducted here is heterogeneous. As such, we chose not to use the funnel plot to assess publication bias (47).

Frequencies and characteristics of adapted, adopted, and novel interventions

We found 523 unique studies fulfilling our inclusion criteria that included enough information for data extraction (Table 2). Of the 523 studies, 21% (k = 111) described adapted interventions, 33% (k = 170) adopted interventions, and 46% (k = 239) novel interventions. Table 2 provides a summary of study characteristics. The most common setting for the studies was within mental health 46% (k = 240). Of the publications, 73% (k = 384) were efficacy studies and 33% (k = 179) effectiveness studies. The majority (87%, k = 454) of the studies were randomized trials. Most control group participants (63%) received another intervention (e.g., treatment as usual or a competing intervention), and the remainder (37%) received no intervention or were on a waiting list. Most studies (55%) had a follow-up period of 12 months or more after baseline (M = 11.9 months, SD = 14.6) and included on the average 164 study subjects. Significant differences between adapted, adopted, and novel intervention studies were found across a range of study characteristics (Table 2).

Table 2

Characteristics of included studies (k = 523) and tests of differences in study characteristics between adapted, adopted, and novel intervention groups.
	Total (k = 523)	Adapted (k = 112)	Adopted (k = 171)	Novel (k = 240)	Significance Tests	p
Setting
Mental Health (%)	46	60	46	40	χ²(2) = 12.36	.01
Social Services (%)	26	17	35	25	χ²(2) = 10.19	.01
Somatic Healthcare (%)	27	23	20	35	χ²(2) = 12.290	.01
Indicated prevention (%)	73	80	71	72	χ²(2) = 2.73	n.s.
Active control group (%)	37	36	30	43	χ²(2) = 7.30	.01
Efficacy trial (%)	73	84	66	74	χ²(2) = 11.90	.05
RCT (%)	87	96	79	89	χ²(2) = 19.48	.0001
Number of months between pre- and follow-up measures (mean)	11.9	11.2	14.2	10.6	F(2,519) = 3.25	.05
Number of subjects (mean)	163.7	138.5	166.8	173.3	F(2,490) = 0.82	n.s

Effect Sizes For Adapted, Adopted And Novel Interventions

As the sample showed significant heterogeneity with a large proportion of random variance, both in the total population (Q(522) = 2489.15, p < .0001), and on the reduced sample (Q(295) = 1589.91, p < .0001) (Table 3). Adapted interventions were found to have the highest standard mean effect size (d = .509), followed by novel interventions (d = .407), and adopted interventions (d = .372). There were significant differences between adapted and adopted interventions, Q(1) = 9.35, p < .002, and between novel and adopted interventions, Q(1) = 5.97, p < .05. No significant difference was found between adapted and novel interventions, Q(1) = 1.08, p > .05.

The results were re-calculated after excluding non-randomized studies and small study populations (less than 50). The result was similar to that of the full sample (Table 3). The effect size was significantly higher in adapted trials compared to adopted, Q(1) = 4,74, p < .05. There were no significant differences between adopted and novel interventions, Q(1) = 1,974, p > .05, or between adapted compared to novel, Q(1) = 1.53, p > .05.

Effect Sizes For Adapted, Adopted And Novel Intervention By Subcategory

Adapted Interventions

For adapted interventions (Q(3) = 14.18, p < .003), pragmatic adaptations had the highest standard mean effect (d = .595), followed by format adaptations (d = .552), target population adaptations (d = .508), and finally cultural adaptations (d = .222) (Table 3). In the analysis of the reduced sample (Q(3) = 12.83, p < .005), the pattern of results was not maintained. Here, format adaptations were found to have a higher standard mean effect (d + = .589) than pragmatic adaptations (d + = .551).

Adopted Interventions

Of the adopted interventions, there was no significant difference in the full sample, Q(1) = 0.35, p > .05, or in the reduced sample analysis, Q(1) = 1.96, p > .05 (Table 3).

Novel Interventions

Out of novel interventions (Q(2) = 21.96, p < .0001), reinterventions had the highest standard mean effect (d = .508), followed by the combined (d = .471), and finally innovations (d = .282). In the reduced sample analysis this pattern of results was maintained Q(2) = 17.95, p < .0001) (Table 3).

Table 3

Results of the Standardized Mean Difference Effects of Main Intervention Types Including Sub-Categories.

Types of Interventions	d	95% confidence intervals	k	d	95% confidence intervals	k
Adapted	.509	.436 ± .582	112	.471	.377 ± .566	58
Cultural	.222	.068 ± .376	11	.191	.023 ± .358	7
Pragmatic	.595	.438 ± .752	33	.551	.328 ± .775	16
Format	.552	.423 ± .681	39	.589	.424 ± .753	21
Target population	.508	.342 ± .674	29	.372	.179 ± .564	14
Adopted	.372	.322 ± .421	171	.341	.271 ± .410	84
Explicit	.376	.297 ± .436	84	.356	.261 ± .452	41
Broadband	.367	.304 ± .449	87	.325	.223 ± .427	43
Novel	.407	.363 ± .451	240	.403	.351 ± .455	154
Innovation	.305	.249 ± .360	114	.282	.216 ± .349	62
Reinvention	.523	.447 ± .599	102	.508	.425 ± .591	77
Combined	.492	.302 ± .682	24	.471	.239 ± .702	15
Total	.414	.384 ± .443	523	.397	.359 ± .435	296
d = Weighted effect size according to random effects model; k = number of studies.

Effect Sizes By Setting

Interventions provided in the mental health setting showed the highest standard mean difference, Q(2) = 95,12, p < .0001 (d = .577), followed by somatic healthcare (d = .361) and last social services (d = .238) (See Table 4).

Table 4

Results of the Standardized Mean Difference Effects of Interventions by Setting.
	Total Study Sample (k = 523)									Excluding non-randomized studies and small study populations (k = 296)
	Mental Health			Social Services			Somatic Healthcare			Mental Health			Social Services			Somatic Healthcare
Type of intervention	d	95% confidence intervals	k	d	95% confidence intervals	k	d	95% confidence intervals	k	d	95% confidence intervals	k	d	95% confidence intervals	k	d	95% confidence intervals	k
Adapted	.662	.542 ± .792	67	.197	.101 ± .293	19	.397	.289 ± .506	26	.660	.488 ± .831	33	.188	.071 ± .304	12	.330	.203 ± .456	13
Adopted	.500	.407 ± .592	79	.269	.204 ± .333	58	.325	.206 ± .444	34	.429	.307 ± .551	39	.261	.177 ± .344	34	.302	.027 ± .577	11
Novel	.587	.501 ± .672	96	.227	.164 ± .289	61	.365	.293 ± .437	83	.601	.496 ± .705	61	.219	.151 ± .286	42	.352	.261 ± .443	51
Total	.577	.522 ± .633	242	.238	.199 ± .278	138	.361	.306 ± .418	143	.564	.491 ± .636	133	.228	.182 ± .275	88	.343	.266 ± .420	75
d = Weighted effect size according to random effects model; k = number of studies.

Because of small sample sizes (< 20) in some cells, it was decided not to do significance tests for differences among the adapted, adopted, and novel interventions separately for mental health, social services, and somatic health.

Moderators Of The Effect Size

The regression analyses, although highly significant, F(6,489) = 9.79, p < .0001, produced a limited combined effect, R²(adj) = .10 (Table 5). Of the 6 predictors, 3 were significant: a higher effect size was present with fewer subjects, a passive control condition, and in studies reporting on an efficacy trial.

Table 5

Multiple regression of effect size for various predictors (k = 489).
	Standardized estimate	p
Intercept	0	.0001
RCT	0.03	n.s.
Number of subjects	-0.16	.001
Indicated prevention	0.05	n.s.
Efficacy trial	0.11	.05
Active control condition	-0.20	.0001
Months follow-up	-0.07	n.s.
Note. Variance inflation factor (VIF) = varied between 1.02 and 1.20

The purpose of this study was to explore the effectiveness of social interventions by replicating and expanding on an earlier meta-analysis comparing novel, adapted, and adopted interventions (36). Our results largely replicate our earlier findings. First, as in the earlier study our results find that novel interventions make up the largest proportion of interventions studied, followed by adopted and adapted interventions. Novel interventions are undoubtedly developed to meet a perceived need in society and this option for intervention delivery is likely preferred when access, awareness, or knowledge of EBIs for specific populations, targeting specific outcomes is scarce. Indeed, although many systematic reviews have identified interventions with strong empirical support for their effectiveness (e.g., 49), systematic reviews also consistently find few to no clearly effective interventions across several population and/or outcome areas (e.g., 50, 51). Second, we find that adapted interventions produced the largest standard mean effect size followed by novel interventions and adopted interventions. This is in line with a body of research reporting successful adaptations to EBIs in more homogeneous samples (52–58). These findings were stable when we controlled for study design and sample size. In addition, the absolute standard mean effect sizes across intervention type found in the current study are higher than those reported in the prior meta-analysis. Despite our finding of differences in standard mean effect sizes across adapted, adopted, and novel categories of interventions, the standard mean effect sizes in all categories fell within the medium range and were significantly different from zero which means on average adapted, adopted, and novel interventions all produce effects in controlled research. However, it should be noted that uncertainty around the point estimate was large for all categories of interventions.

In addition, we aimed to expand the prior meta-analysis by comparing the effectiveness of these groups of interventions in various types of settings. We found the same pattern of results in the standard mean effects in mental health and somatic healthcare settings. Although these results were stable when controlling for study design and sample size within mental health, they were not stable within somatic healthcare. This, however, may be due to the small number of studies within the adapted and adopted categories in these analyses. Studies conducted within the social services showed a different pattern of results. In the social service setting, adopted interventions evidenced a higher standard mean effect, followed by novel interventions. Adapted interventions had the smallest mean effect within a social service setting. These results were stable when we controlled for study design and sample size. This result may be due to the relatively low number of adapted interventions within the social work setting found in this study. It may, however, be related to specific characteristics of the interventions, delivery, or research environment within the social services context (41, 59). The extent to which there are any specific characteristics of social service interventions or the social services setting that might impact intervention effectiveness needs to be further investigated.

As in the prior meta-analysis, we investigated subgroups of adapted, adopted, and novel interventions although we have refined and expanded on the range of subgroups in the current study. Interestingly, within adopted interventions, cultural adaptations (i.e., adaptations made to the intervention to suit cultural norms) were found to have the fewest number of studies within and across subgroups as well as the smallest standard mean effect size within and across subgroups. This is interesting as this area of adaptation is arguably the most prominent within the literature on intervention adaptation (e.g., 24, 34, 60–63). It should be noted, however, that much of the extant literature on cultural adaptation addresses cultural adaptations to EBIs for use with minority populations within the same context as EBIs were developed as opposed to cross-cultural adaptations (i.e., when EBIs are transported to new cultural contexts such as countries) which is what we report in the current study. This type of transfer introduces not only changes in the cultural norms of target populations but can also mean changes in broad societal norms and how services are delivered. Similarly, within the novel subgroup, combined interventions (see Table 1) were relatively few and innovative interventions exhibited the lowest standard mean effect size within the novel subgroup. As these subgroups represent a refinement and expansion of coding definitions, we are unable to compare these results to the prior meta-analysis. However, across subgroups the results of the current study find standard mean effect sizes to be higher than that of the prior meta-analysis as well as exhibiting more variability in standard mean effect sizes across subgroups.

Methodological Discussion

Although this study has several strengths there are also four main limitations that should be highlighted. First, although intensive work was done to develop and agree upon the coding manual, the boundaries between the subcategories were not always clear-cut. This introduces not only a methodological challenge but also reflects the complexity of intervention development, evaluation, and spread. The ability to conduct valid and reliable meta-analyses is constrained by the information contained in the report of original research (for a short discussion see e.g., 64), however, the intervention development and/or adaptation process may not always be clearly described in publications describing research on interventions. It is a common finding that despite methodological guidance regarding the inclusion of clear descriptions of interventions assessed in evaluative research (e.g., 65), investigations and reviews on the effectiveness of interventions consistently point to shortcomings in the extent to which interventions are developed and described (e.g., 66, 67). Despite this, the interrater reliability between raters was high. Second, we did not conduct a risk-of-bias assessment (48, 68). Therefore, the methodological quality across categories (adapt, adopt, novel) and studies may differ which would impact the results presented here. Third, we did not assess the extent to which within study fidelity was tracked or maintained. Although our analysis includes adapted interventions, it should be stressed that these represent intentional a priori efforts to adapt EBIs prior to implementation (as opposed to therapist or program drift). Once these intentional adaptations are made the adapted interventions need still be implemented with fidelity. This is the case for all categories of intervention assessed in this meta-analytic review. We are, however, unable to assess the extent to which the adapted, adopted, and novel interventions included were implemented with fidelity within each study. Finally, it should be noted that our analysis includes research conducted in Sweden only. The extent to which these findings are generalizable to other contexts and settings is unknown.

The research to practice pathway is based on the idea that interventions are developed and tested and used in the way they were designed. The results reported here are in line with a growing body of evidence suggesting that there is a need to take the fit between the EBI and the context into account when implementing interventions, as shown in the superiority of adapted EBIs and interventions specifically developed for a certain context. This complicates the accumulation of knowledge across studies, with implications for how primary studies as well as reviews of intervention studies are conducted. Primary studies that illuminate what works for whom and how, that is, measurement of the moderators and mediators of intervention effects that are adapted and measured across studies will be vital for ensuring the possibility for an accumulation of knowledge across studies. This type of measurement is also important for increasing our understanding of the core components of interventions. Review methods that can take the variation between cases into account are called for. In conclusion, we find that standardized mean effects for interventions across categories are significantly different than zero, indicating that adapted, adopted and novel programs all provide benefit to service users. Standardized mean effects for adapted interventions were, however, larger than that for novel and adopted programs and smallest for adopted programs.

ACT Acceptance and Commitment Therapy

BRÅ Swedish Crime Victim Authority

EBI Evidence-based intervention

EBSCO Elton B. Stephens CO

ERC European Research Council

FORMAS Swedish Research Council for Sustainable Development

FORTE Swedish Research Council for Health Working Life and Welfare

ISRCTN International Standard Randomised Controlled Trial Number

RCT Randomized controlled trial

VINNOVA Swedish Innovation Agency

VR Swedish Research Council

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests financial or otherwise.

Funding

This study was funded by two grants from the Swedish Research Council for Health, Working Life and Welfare (Forte; Dnr: 2019-01737 and 2018-01315) as well as a grant from the Swedish Research Council (Dnr: 2016-01261).

Authors contributions

TO Investigation, data curation, writing original draft, review, and editing, funding acquisition. UTS Writing original draft, review, editing and funding acquisition. HH Writing original draft, review, editing and funding acquisition. EV Formal analysis, writing original draft, review, and editing. KS Conceptualization, methodology, formal analysis, investigation, data curation, writing original draft, review, and editing, funding acquisition. All authors read and approved the final manuscript.

Acknowledgements

Not applicable.

Sundell K, Olsson TM. Social Intervention Research. Oxford Bibliographies. 2017.
Fraser MW, Galinsky MJ. Steps in intervention research: designing and developing social programs. Res Social Work Pract. 2010;20(5):459–66.
Fraser MW, Richman JM, Galinsky MJ, Day SH. Intervention Research: Developing Social Programs. Tripodi T, editor. New York: Oxford University Press; 2009.
Duncan E, O’Cathain A, Rousseau n, Croot L, Sworn K, Turner KM, et al. Guidance for reporting intervention development studies in health research (GUIDED): an evidence-based consensus study. BMJ OPen. 2020;10:e033516.
Flay BR, Biglan A, Boruch RF, Gonzalez Castro F, Gottredson D, Kellam SG, et al. Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci. 2005;6(3):151–75.
O'Connell ME, Boat T, Warner KE, editors. Preventing mental, emotional, and behavioral disorders among young people: progress and possibilities. Washington, D. C.: The National Academies Press; 2009.
Eichas K, Ferrer-Wreder L, Olsson TM. Contributions of positive youth development to intervention science. Child & Youth Care Forum. 2019;48(Special Issue Positive Youth Development Across Cultures):279 – 87..
Mrazek PJ, Haggerty R, editors. Reducing risks for mental disorders: frontiers for preventive intervention research. Washington, D. C.: National Academy Press; 1994.
Gottfredson DC, Cook TD, Gradner FEM, Gorman-Smith D, Howe GW, Sandler IN, et al. Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: next generation. Prevenion Science. 2015:34.
Aarons GA, Sklar M, Mustanski B, Benbow N, Brown CH. ”Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implement Sci. 2017;12(1):111.
Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prevenion Sci. 2004;5(1):47–53.
California Department of Social Services. CEBC - California Evidence-Based Clearinghouse for Child Welfare. 2006.
Arnold Ventures Evidence-Based Policy Team. Social Programs that Work. 2022.
Dymnicki AB, Wandersman A, Osher DM, Pakstis A. Bringing interventions to scale: implications and challenges for the field of community psychology. In: Bond MA, Serrano-Garcia I, Keys CB, Shinn M, editors. APA handbook of community psychology: methods for community research and action for diverse groups and issues. American Psychological Association; 2017. pp. 297–310.
von Thiele Schwarz U, Aarons GA, Hasson H. The value equation: three complementary propositions for reconciling fidelity and adaptation in evidence-based practice implementatin. BMC Health Serv Res. 2019;19:868.
Chambers DA, Glasgow RE, Strange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(1):117.
Baumann A, Stirman LJC. SW. Adaptation in dissemination and implementation science. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. 2. Oxford: Oxford University Press; 2017. pp. 286–300.
Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front Public Health. 2018;6:102.
Kilbourne AM, Neumann MS, Pincus H, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci. 2007;2(1):42.
Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.
Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5:67.
Bonell C, Prost A, Melendez-Torres GJ, Davey C, Hargreaves JR. Will it work here? A realist aöpproach to local decisions about implementing interventions evaluated as effective elsewhere. J Epidemiol Commun Health. 2021;75(1):46–50.
Moore G, Campbell M, Copeland L, Craig P, Movsisyan A, Hoddinott P, et al. Adapting interventions to new contexts-the ADAPT guidance. BMJ. 2021;374(1679):1–10.
Ferrer-Wreder L, Sundell K, Mansoory S. Tinkering with perfection: Theory development in the intervention cultural adaptation field. Child and Youth Care Forum. 2012;41:149–71.
Moore J, Bumbarger B, Cooper B. Examining adaptations of evidence-based programs in natural contexts. J Prim Prev. 2013;34(3):147–61.
Escoffery C, Lebow-Skelley E, Haardoerfer R, Boing E, Udelson H, Wood R, et al. A systematic review of adaptations of evidence-based public health interventions globally. Implement Sci. 2018;13(1):125.
Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14:58.
Lee SJ, Altaschul I, Mowbray CT. Using planned adaptation to implement evidence-based programs with new populations. Am J Community Psychol. 2008;41(3–4):290–303.
Ahmad A, Larsson B, Sundelin-Wahlsten V. EMDR treatment for children with PTSD: results of a randomized controlled trial. Nord J Psychiatry. 2007;61(5):349–54.
Calbring PER, Gunnarsdóttir M, Hedensjö L, Andersson G, Ekselius L, Furmark T. Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioral therapy with telephone support. Br J Psychiatry. 2007;190(2):123–8.
Livheim F, Hayes L, Ghaderi A, Magnusdottir T, Högfeldt A, Rowse J, et al. The effectiveness of Acceptance and Commitment Therapy for adolescent mental health: Swedish and Australian pilot outcomes. J Child Fam stud. 2015;24(4):1016–30.
Castro F, Barrera M Jr, Holleran Steiker LK. Issues and challenges in design of culturally adapted evidence-based interventions. Annual Reviews of Clinical Psychology. 2010;6:213–39.
Resnicow K, Soler R, Braithwaite RL, Ahluwalia JS, Butler J. Cultural sensitivity in substance use prevention. J Community Psychol. 2000;28(3):271–90.
Barrera M, Berkel C, Castro FG. Directions for the advancement of culturally adapted preventive interventions: local adaptations, engagement, and sustainability. Prev Sci. 2017;18(6):640–8.
Kling Å, Forster M, Sundell K, Melin L. A randomized controlled effectiveness trial of Parent Management Training with varying degrees of therapist support. Behav Ther. 2010;41:530–42.
Sundell K, Beelmann A, Hasson H, von Thiele Schwarz U. Novel programs, international adoptions, or contextual adaptations? Meta-analytical resurlts from German and Swedish intervention research. Journal of Clinical Child & Adolescent Psychology. 2015;pre-print:1–13.
Bergström M, Sundell K, Olsson TM, Leander L, Åström T. Interventions in child welfare - a Swedish inventory. Child & Family Social Work. 2022;Early View:1–8.
Liedgren P, Kullberg C. ’Easy ride or born to be wild’? The traveling of evidence-based social owrk to Sweden. Eur J Social Work. 2022;25(2):224–37.
Sundell K, Åhsberg E. Trends in methodological quality in controlled trials of psychological and social interventions. Res Social Work Pract. 2018;28(5):568–76.
Kugley S, Wade A, Thomas J, Mahood Q, Jorgensen A-MK, Hammerstrom K, et al. Searching for studies: a guide to information retrieval for Campbell Systematic Reviews. The Campbell Collaboration; 2015.
Sundell K, Olsson TM. Svenska effektutvärderingar av beteendemässiga, psykologiska och sociala insatser. Stockholm, Sweden: the Swedish Research Council for Health. Working Life, and Welfare (FORTE); 2021.
Sundell K, Åhsberg E. Trends in methodological quality in controlled trials of psychological and social interventions. Res Social Work Pract. 2016;28(5):568–76.
Lipsey MW, Wilson D. Practical Meta-Analysis (applied social research methods). Thousand Oaks: Sage; 2000.
Cook RD. Detection of influential observation in linear regression. Technometrics. 1977;19(1):15–8.
Walfish S. A review of statistical outlier methods. Pharm Technol. 2006;30(11):82.
Mason CH, Perreault Jr. WD. Collinearity, power, and interpreatation of multiple regression analysis. J Mark Res. 1991;28(3):268–80.
Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleadning funnel plot. BMJ. 2006;333(7568):597–600.
Higgins JPT, Green S. The Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration; 2013.
Fordham B, Sugavanam T, Edwards K, Stallard P, Howard R, das Nair R, et al. The evidence for cognitive behavioural therapy in any condition, population, or context: a meta-review of systematic reviews and panoramic meta-analysis. Psychol Med. 2021;51(1):21–9.
Olsson T, Långström N, Skoog T, Andrée Löfholm C, Leander L, Brolund A, et al. Systematic review and meta-analysis of noninstitutional psychosocial interventions to prevent juvenile criminal recidivism. J Consult Clin Psychol. 2021;89(6):514–27.
Petersen K, D RC, Weisburd D, Taylor B. Effects of second responder programs on repeat incidents of family abuse: an updated systematic review and meta-analysis. Campbell Syst Reviews. 2022;18(1):e1217.
Gladstone TRG, Rintell LS, Buchholz KR, Myers TL. Adaptation of an evidence-based online depression prevention interventin for college students: intervention development and pilot study results. Social Sciences. 2021;10(10).
Rajabiun S, Myers J, Goldberg J, Fortu K, Sullivan M, Berthaud V, et al. Outcomes from the adaptation of an evidence-based peer linkage and re-engagement intervention for women of color with HIV in three urban clinics. AIDS Behav. 2021;26:415–24.
Stahmer AC, Rieth SR, Dickson KS, Feder J, Burgeson M, Searcy K, et al. Project ImPACT for Toddlers: Pilot outcomes of a community adaptation of an intervention for autism risk. Autism. 2019;24(3):617–32.
Stanton B, Guo J, Cottrell L, Galbraith J, Li X, Gibson C, et al. The complex business of adapting effective interventions to new populations: an urban to rural transfer. J Adolesc Behav. 2005;37(2):163.
Michelson D, Davenport C, Dretzke J, Barlow J, Day C. Do evidence-based interventions work when tested in the ”real world?” A systematic review and meta-analysis of parent management training for the treatment of child disruptive behavior. Clin Child Fam Psychol Rev. 2013;16:18–34.
Galovski TE, Blain LM, Mott JM, Elwood L, Houle T. Manualized therapy for PTSD: Flexing the structure of cognitive processing therapy. J Consult Clin Psychol. 2012;80:968–81.
Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and met-analysis. World Psychiatry. 2014;13:288–95.
Olsson TM, Sundell K. Research that guides practice: outcome research in Swedish PhD theses across seven disciplines 1997–2012. Prev Sci. 2016;17(4):525–32.
Bernal G, Jimenez-Chafey MI, Domenech Rodriguez MM. Cultural adaptation of treatments: a resource for considering culture in evidence-based practice. Prof Psychology: Res Pract. 2009;40(4):361–8.
Gonzales NA. Expanding the cultural adaptation framework for population-level impact. Prevenion Sci. 2017;18:689–93.
Mejia A, Leijten P, Lachman JM, Parra-Cardona J. Different strokes for different folks? Contrasting appraoches to cultural adaptation of parenting interventions. Prev Sci. 2017;18:630–9.
Marsiglia FF, Booth JM. Cultural adaptation of interventions in real practice settings. Res Social Work Pract. 2015;25(4):423–32.
Olsson T, Sundell K, Leander L. The devil is in the details: unpacking the systematic in systematic reviews: short reply to reconsidering the effectiveness of mentoring for prevention of juvenile criminal recidivism. J Consult Clin Psychol. 2022;90(8):652–4.
Schulz KF, Altman DG, DM, the CONSORT Group. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine. 2010;8(18):1–9.
Glasziou P, Meats E, Shepperd S. What is missing from descriptions of treatment in trials and reviews? BMJ. 2008;336:1472–4.
Maden M, Cunliffe A, McMahonb N, Booth A, Carey GM, Paisley S, et al. Use of programme theory to understand the differential effects of interventions across socio-economic groups in systematic reviews-a systematic methodology review. Systematic Reviews. 2017;6(266).
Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration and John Wiley & Sons; 2008.

Download PDF

Version 1

posted

You are reading this latest preprint version

Adapted and novel interventions are more effective than adopted interventions: a meta-analytic replication of controlled social intervention research across prevention level and setting

Status:

Version 1

Abstract

Contributions To The Literature

Background

Method

Design and setting

Eligibility Criteria

Inclusion criteria

Exclusion criteria

Information Sources

Search Strategy

Data Items And Data Extraction

Study characteristics

Included intervention

Intervention type

Outcome Measures

Statistical Procedures

Sensitivity Analysis

Results

Frequencies and characteristics of adapted, adopted, and novel interventions

Effect Sizes For Adapted, Adopted And Novel Interventions

Effect Sizes For Adapted, Adopted And Novel Intervention By Subcategory

Adapted Interventions

Adopted Interventions

Novel Interventions

Effect Sizes By Setting

Moderators Of The Effect Size

Discussion

Methodological Discussion

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1