Rationale for a Rapid Realist Review
Traditional approaches to literature reviews (systematic reviews and meta-analyses) assume outcomes are generated by linear causation [32]. While these approaches work well for studies conducted with highly controlled settings and exposures (e.g. randomized control trials); they severely limit our understanding of complex and pragmatic interventions [33]. Complex and pragmatic interventions require methods that offer a more comprehensive explanation of the ‘process’ that was undertaken [34]. Therefore, a realist synthesis is well-suited to meet these needs as it is uses a theory-driven approach to synthesize complex evidence from diverse sources and provide an understanding of why and how complex interventions works [28, 29].
Specifically, a realist synthesis aims to understand how, for whom, where, and why the intervention is effective or ineffective [28, 29]. This is accomplished by examining the “mechanisms”, exploring the “contexts” where the intervention occurred, and then linking these contexts and mechanisms to the “outcome” of the intervention [28]. As per the realist definition, mechanisms are the “underlying entities, processes, or [social] structures which operate in particular contexts to generate the outcomes of interest” [35]. This combination of the context (C), mechanisms (M), and outcome (O) in an intervention is called a C-M-O configuration. Recurrent patterns of C-M-O configurations are known as demi-regularities, or semi-predictable pattern/pathway of how a program functions. In other words, demi-regularities are a broad rule for how and when certain outcomes usually occur [28].
While full realist reviews can require a considerable dedication of time to the exploration of literature and subsequent analysis, rapid realist reviews (RRRs) have been used to enable a quicker transition from research to policy and/or practice [36]. Given the need for a timely synthesis and its application for the Picking Up the PACE programme, we undertook a rapid realist review; which allows us to maintain the core elements of the realist methodology and produce timely data.
Prior to this rapid realist review, a pre-specified protocol was registered (PROSPERO registration number: CRD42017064430) and published [37] which included the research question, search strategy, synthesis methodology, preliminary program theory, definitions, inclusion criteria for relevance screening, data extraction form, quality assessment tool, and plans for dissemination. An overview of the methods and any modifications to the original protocol are described below. Utilizing the RAMESES (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) [38], and adapting it to follow a rapid realist review [34], the following steps were applied:
Clarifying the Scope
Identifying the research question. This rapid realist review supports a larger program, Picking Up the PACE, that aims to increase the ability of healthcare providers to offer evidence-based interventions to tobacco users which encompass changing modifiable risk behaviours (excess alcohol consumption, physical inactivity, poor diet, stress, and poor sleep) to ultimately achieve long-term smoking abstinence. As a result, this review focuses on smoking cessation in the context of multiple health behaviour change interventions that also address these other risk behaviours.
In order to clarify the scope of the rapid realist review, a multidisciplinary team with expertise in knowledge synthesis, public health, and multiple health behaviour change met in-person on nine occasions for 1 hour over the course of six months. Our initial research question was: “What factors are associated with effective multiple health behaviour change (three or more behaviours including smoking)?”
Changes in the rapid realist review process. After a preliminary review of the data, further specificity of the study question was required to meet the desired outcome. The contexts and mechanisms involved in changing multiple health behaviours might be different than those involved in smoking cessation. Thus we modified our research question to: “What contexts and mechanisms are associated with improving smoking cessation outcome in interventions that target two or more additional unhealthy behaviours.”
Initial theory. We identified our initial theory of how, when, and why multiple health behaviour change interventions work by reviewing seven large-scale multi-factorial cardiovascular disease and cancer risk interventions [39-45]. These studies included the Multiple Risk Factor Intervention Trial (MRFIT) [39], the North Karelia Project [40], the Stanford Five City Project [41], Project PREVENT [43], the Minnesota Heart Health Program [45], the Mediterranean Lifestyle Trial [44], and the BETTER Trial [42], all of which are well-known studies that promoted multiple health behaviour change in large community samples [46]. As specified in our protocol manuscript [37], our preliminary review of these seven interventions involved having two independent reviewers extract the following information from the studies:
1. The specific activities within each intervention. Activities are physical/tangible tasks that were undertaken by the intervention (e.g. counselling, sharing of educational flyers, workshop, courses, prize draw). Please note, we coded all activities undertaken by intervention for any behaviour, not only those related to smoking.
2. The setting in which the intervention took place, including physical environment, social setting, and political climate (if provided).
3. The outcomes of each intervention, including any behavioural and/or clinical outcomes.
Through this preliminary review, we found that successful interventions usually had: pre-existing infrastructure that facilitates the delivery of the intervention, and targeting regions (e.g. geographic, population groups) where the need for the intervention is well-characterized. Furthermore, activities undertaken by these interventions often targeted the surrounding community and/or organizational structure. This multi-level approach appears to be in an effort to change the physical and social opportunities that can help facilitate multiple health behaviour change in individuals. Individual-level activities frequently focused on increasing patient’s awareness and knowledge, improving feelings of support, empowerment, and incorporating incentives for completing activities.
Upon closer review, we realized that these activities mapped onto the COM-B model; which stipulates that behaviour change requires change in one or more of the following component: capability, opportunity, and motivation [30]. All seven studies used in developing our initial program theory sought to change at least one component of this behavioural system. We used the taxonomy of behaviour change techniques [31] to code each activity specified in the studies and we cross-referenced these codes with the COM-B model. We used Table 2 in Michie and colleagues’ article to help us create the links between the components of the COM-B model and the BCT taxonomy [30]. For example an intervention that helped participants set a quit date was categorized as BCT 1.3“Goal setting” and consequently coded under “Capability” within the COM-B model. A visual depiction of this theory can be found in the published protocol [37].
The coded data was reviewed by our expert panel, which had a total of 11 members and was comprised of representatives from the Medical Psychiatry Alliance, Public Health Ontario, and the Centre for Addiction and Mental Health. Over the course of 9 in-person meetings, the expert panel assisted the research team with the review and development of the initial program theory.
Searching for Relevant Evidence: Search Strategy and Eligibility Criteria
To test our program theory, a search strategy was developed and implemented to retrieve relevant primary data from both academic and grey literature. The search strategy was informed by the research team and developed by a medical librarian who executed the search across multiple bibliographic databases [37]. After our protocol was published, minor changes were made to the search strategy (see Additional File 1). The initial search aimed to identify as many multiple health behaviour interventions as possible, allowing the team to accurately identify trends across the literature.
To identify grey literature from Canada, Europe, and the USA, variations of the phrase “multiple health behaviours” were used to hand search the websites and online repositories of international, national, and provincial health organizations, health behaviour/condition-specific associations, clinical trial registries, and grey literature repositories. Reference lists of three systematic reviews and meta-analyses [27, 47, 48] were also hand searched to identify any relevant resources not captured by the systematic searches. No additional articles were included from the grey literature or reference list searches. After the search was complete, we chose to exclude books and reviews. It should be noted that 22 interventions were identified as having more than one publication reporting similar results. In these cases, the lead scientist and two additional members of the team selected one article per intervention to represent the contexts, mechanisms, and outcomes of the intervention, and excluded other articles associated with that intervention.
Relevance Confirmation, Data Extraction, and Quality Assessment
Two independent reviewers assessed each study to determine its relevance to our research question, extract pertinent information, and appraise its quality using pre-designed and pre-tested relevance screening and data extraction forms [37]. The systematic review software DistillerSR [49] was used for this process. As described in our protocol [37], to be included in this review the study had to:
· Describe interventions that targeted tobacco use as well as two or additional modifiable risk behaviours (excess alcohol consumption, physical inactivity, poor diet, stress, and poor sleep).
· Report on long-term (i.e. follow-up at 5 months or longer) smoking cessation outcomes
Since we had multiple study designs included in this rapid realist review, a combination of the Mixed Methods Appraisal Tool (MMAT) [50] and the Critical Appraisal Skills Programme (CASP) [51] was used to evaluate the methodological quality of qualitative, quantitative, and mixed method studies. Each type of study was assessed by two reviewers using a pre-designed quality assessment form [37]. As per MMAT and CASP appraisal methods, the quality criteria differed based on the study design; quantitative – randomized controlled trial (eight criteria), quantitative – non-randomized (10 criteria), qualitative (10 criteria), and mixed-method (six criteria). These criteria were scored using a nominal scale (Yes/No/Can’t Tell).
Based on the scoring metrics of the MMAT [50] and adjusting for the additional CASP criteria [51], an overall quality score was calculated for each study using the descriptors *, **, ***, and ****. For all types of studies, the score was derived by taking the number of criteria met and dividing it by the number of criteria. Scores were assigned the following descriptors: 0-25% (*), 26-50% (**), 51-75% (***), and 76%+ (****). To score the mixed methods studies, the overall quality could not exceed the quality of the weakest section of the study. For example, in a mixed method study, if the qualitative score is (**), and the quantitative and mixed method scores are both (***), the study is assigned an overall score of the lowest section (**).The questions used to score each study can be found in Additional File 2.
Prior to data extraction and coding of the context, mechanisms, and outcomes within the studies, reviewers were trained on the COM-B model, the Behaviour Change Wheel and the BCT taxonomy [30, 31]. They were also trained on how to characterize the various techniques that are used within interventions and map these techniques onto the COM-B model. Once trained, the following process was also undertaken by the two independent reviewers:
· Review article to identify and record the activities that took place in the intervention.
· Code the modifiable risk behaviours the intervention was targeting.
· Code which techniques were applied to each activity, as defined by the BCT taxonomy. Please note, we coded all activities and corresponding techniques in the intervention, not only those that are specific to smoking cessation.
· Determine how each technique is associated with the COM-B model.
· Code the target population (e.g. gender, ethnicity, general public vs patients)
· Code the smoking outcome(s), including whether there was a statistically significant change and the follow-up period in which the outcome was assessed (e.g. end of treatment, 6 months, 12 months, or 24 months)
· Code the context where the intervention took place (e.g. region, clinical setting, clinical, community-based settings, and/or school-based settings)
At each step, discrepancies between two reviewers were resolved by consensus or, when necessary, by a third reviewer.
Data Analysis and Synthesis Process
The data from DistillerSR [49] was exported to Microsoft Excel for descriptive analysis and analyzed using NVivo 11 [52]. To determine whether the intervention fit the initial program theory and to identify if there were any emerging patterns in the types and combination of C-M-O’s configurations used, the reviewers examined the studies to see whether the interventions had also focused on changed physical and social opportunities and/or other behaviour change techniques such as raising awareness, increasing knowledge, and encouraging empowerment. The behaviour change techniques did not have to be specific to smoking and could be targeting any modifiable risk behaviours. For example, if a multiple health behaviour change intervention offered membership to a gym to help improve physical activity, this was coded as Opportunity.
Smoking cessation outcomes were measured in a variety of ways across articles, including different time points (e.g. at end of treatment, three months, 12 months), duration of abstinence (e.g. 7-day point prevalence abstinence vs last 30 days), and presentation of data (e.g. descriptive vs statistical analyses). These outcomes were verified (e.g. biochemically) or were self-reported. The literature shows that both types of outcomes are valid [53-55] and therefore we did not differentiate between self-reported measures and biochemically verified measures. However, studies that used non-validated measures/screeners for self-report questions would be penalized in the quality assessment score. Please see Additional File 2 questions used to score each study.\
As a result, we organized our findings by whether statistically significant smoking cessation outcomes were observed and whether the outcome was measured long-term (i.e. ≥5 months). Within these outcome types, the interventions were organized by the three categories that were then used to identify the mechanisms (capability, opportunity, and motivation) and the context in which the intervention occurred. Many of the reviewed articles did not describe the context in which the intervention was implemented in sufficient detail. Thus we decided to be as broad as possible and divided context into three categories: 1) the continent in which the intervention took place, 2) the type of setting (e.g. clinical, workplace) and 3) whether it was a multidisciplinary intervention. We established the following criteria to report demi-regularities:
· There were a minimum of three interventions using the specific C-M-O configuration.
· Among interventions with a specific C-M-O configuration, either ≥60% OR ≤40% of these interventions reported statistically significant increase in smoking cessation.
To present an example of how this process works, if we discover a C-M-O configuration (e.g. Clinical Setting – Capability – Smoking Cessation Outcome) within an intervention, there must be at least two other interventions with this C-M-O configuration to allow for further analysis. In this hypothetical example, if we have a total of ten interventions that have ‘Clinical Setting-Capability-Smoking Cessation Outcome’ configuration, we then have to determine what percentage of these studies reported a statistically significant increase in smoking cessation. In order for this C-M-O configuration to be categorized as a demi-regularity, at least 60% of these interventions must report a statistically significant increase in long-term smoking cessation outcome (i.e. ≥5 months). The demi-regularity in this case would be that interventions in clinical settings that target capability are more likely to lead to improvement in smoking cessation outcome. Alternatively, if ≤40% of the interventions reported a statistically significant increase in smoking cessation outcome, the demi-regularity would suggest that interventions in clinical settings that target capability are less likely to lead to improvements in smoking cessation outcomes.
In this paper, we analyzed demi-regularities in interventions that were rated four stars in our quality rating, used statistical analyses, and reported long-term smoking cessation outcomes (i.e. ≥5 months). We chose to only include those interventions with a four star rating as they have the least amount of bias. Once a demi-regularity was discovered, studies that had lower quality assessment scores (less than four stars), and/or did not perform statistical analyses were included in our pool for analysis to confirm if the previously observed demi-regularity persisted.