Designing and qualifying the theory in theory-driven evaluation (TDE)
A theory has been defined as “a set of analytical principles or statements designed to structure our observation, understanding, and explanation of the world.”(16) This definition is broad and can generate confusion. Attempting to clarify this definition in the science implementation field, Nilsen (16) proposed three main conceptualizations: a theory can be described as explanatory, “made up of definitions of variables (…) and a set of relationships between the variables and specific predictions“; a model can be described as descriptive, not explanatory, and as providing a “deliberate simplification of a phenomenon or a specific aspect of a phenomenon”; or it can be seen as a framework, which is also descriptive, but not explanatory, “presenting categories (e.g., concepts, constructs, or variables) and the relations between them that are presumed to account for a phenomenon”. Nilsen describes five theoretical approaches involved in implementation science: the process models, the determinant frameworks, the classic theories, the implementation theories, and the evaluation frameworks (16). The differences among these definitions remain slight, and some cross-over exists between them. For example, some of the “classic theories” (i.e., essentially psycho-social theories) are called “models”, despite their offering explanations, as the health belief model (17). Another example would be the evaluation frameworks (e.g., theory-driven evaluation), which describe steps by which to proceed with an evaluation. These can also be defined as process models dedicated to evaluation.
In this context, it is not easy to clarify the theory in theory-driven evaluation; is it one of these theories, models, or frameworks? A combination of them? In TDE, the theory explains how the program produces its effects (why and how the intervention works) by defining a set of explicit or implicit assumptions on the part of stakeholders about what action is required to solve a problem and why the problem will respond to this action (9,18,19). In the line of our previous work on interventional systems, assuming this blurring between context and intervention components (5,20), the theory in TDE should integrate various elements coming from other theories, models, and frameworks because i) it is explanatory, considering what causal pathway is expected to meet the goal, similar to classical theories; ii) it hypothesizes which specific actions and sequences of implementation contribute to this causal pathway, similar to a process model; and iii) it considers contextual elements and their influence within a specific setting. To be clear, we adopted Moore and Evans’ statement (21) : “[A] single theory will not tell the whole story because it could place weight on some aspects (e.g., certain causal factors) at the expense of others.” According to this perspective, excluding one theoretical approach in favor of another could provide a partial understanding of the intervention system; the theory in theory-driven evaluation should consider all of these theoretical approaches, as long as they are well understood and differentiated. Therefore, we suggest introducing the concept of interventional system theory(ISyT), which could combine that we have defined as:
- Causal theory (the term “theory” was chosen because of its explanatory aspect and its independence of a classic theory): Causal theory involves explanatory and mechanistic components, but it also considers all of the determinants likely to be involved as barriers or enablers to meet goals, as well as actions triggering the expected mechanisms.
- Action model (the term model was chosen because of its sequential pattern, as in a process model): Action models are employed by developers and implementers. They provide concrete elements of action and implementation intended to guide the process to meet the purposes. The core aspect of the action model is its focus not only on the activities involved in outcomes but also on the sequences, the resources, and the prerequisites needed to implement them.
Figure 1 describes interventional system theory (Figure 1. Interventional system theory (ISyT)).
Articulation of various theoretical approaces with interventional sytem theory
Once the interventional system theory is defined, the question of its articulation with existing theories, models, and frameworks arises. Actually, we suggest that all of its components could be informed by those theories a priori, similar to a grounded theory (i.e., grounded in the context). Indeed, the causal theory could be informed by explanatory theory(ies) such as classic theory(ies) (e.g., mechanisms and causal relationships between variables) or by determinants frameworks (contextual influencing parameters), and the action model by implementation theories or process models.
For example, one of the mechanisms involved in behavioral change is motivation (a mechanism), which is enhanced by self-efficacy (another mechanism). Motivation and self-efficacy and their influence on behavior (goal) have been documented in numerous classic theories. One way to increase self-efficacy is to provide positive feedback on the change process (interventional component), an approach that has been documented by numerous implementation theories. This positive feedback could be provided by professionals (another interventional component) but also by relatives or communities around the person, who need to be involved and sensitized to support the person in change processes (another interventional component). Some experiments have described how to mobilize these communities according to specific or generalizable process models that describe training or supportive processes. The ability to do this could be dependent on multiple contextual elements that may act as barriers or enablers (contextual elements). For example, the motivation to change could be impeded or favored by the opportunity for change due to a lack of, or the provision of, resources to support the change. The roles of these contextual factors have been documented by numerous socioecological determination frameworks (22–25).
Hence, we attempt to synthesize these different approaches (farmeworks model, classic theories, etc.) in the population health intervention field to assess how they might help to clarify theoretical understanding of the intervention system. This work is summarized in Table 1 (Table 1 - Four theoretical approaches for understanding the population health interventional system (ISy)), which clarifies these theories’ contributions to an understanding of the interventional system, indeed, the conception of intervention sytem theory.
The role of mechanisms in interventional system theory
The aim in conceiving this ISyT is to understand how these mechanisms are produced (5,6) and under which conditions they trigger particular results. These mechanisms are consequently considered the key functions of the interventional system (5,6). Their integrity guarantees the transferability of an intervention. We should distinguish these key functions from their form, which reflects adaptation to the context. Note that different definitions of mechanisms exist (5,30). Machamer et al. (31) defined them as “Entities and activities organized such that they are productive of regular changes from start or set-up to finish or termination of conditions.” In the realistic approach, a mechanism is “an element of reasoning and reaction of an agent with regard to an intervention productive of an outcome in a given context” (30). In health psychology, they can be defined as “the processes by which a behavior change technique regulates behavior” (32). This could include, for instance, how practitioners perceive an intervention’s usefulness or how individuals perceive their ability to change their behavior.
These definitions have a common point: mechanisms are the inescapable prerequisites for change. In this respect, we argue that they are key features for investigation by means of a TDE, and they are the elements of an interventional system that must be reproduced during transfer to other settings (5,6). Indeed, many combinations of intervention–contextual elements could produce the same mechanism (e.g., some people are sensitive to emotional support during smoking cessation, whereas others would prefer to have technical support, but both types of support could trigger the same mechanism: feelings of being reassured, helped, and supported). During the transfer to other settings, implementation variations, population characteristics, and organizational elements could change, producing the same mechanism or mechanisms different from those expected. The intervention process could be adapted to each new context if these adaptations allow the production of the expected mechanisms. This is why the mechanisms are the key functions that must be reproduced.
Therefore, the characteristics described in Table 2 can be attributed to interventional system theory (ISyT).
Using interventional system theory (ISyT) in the evaluation process
With reference to the theory-driven paradigm, interventional system theory contributes to the design and evaluation of an interventional system. As in the various theory-driven frameworks, e.g., realistic evaluation (33) or the theory of change (TOC) framework (10), the process follows different major steps in articulating definitions, revising theory, and collecting data to inform and refine the theory.
To be complete and effective, this process i) should be established in a participatory way that combines experiential and scientific knowledge and involves different stakeholders, including populations targeted by the intervention, field professionals developing the intervention who are aware of the context, and researchers providing a global and multidisciplinary analysis of the studied phenomenon; ii) employing an evidence-based and rigorous consensus building process that includes, for example, literature reviews, workshops, and exploratory studies; iii) using a hybrid approach that combines both hypothetico-deductive and inductive methods; and iv) mobilizing quantitative and qualitative data collection in mixed-methods designs (34,35). This process could be defined as depicted in Figure 2 (Figure 2 – Using ISyT in the evaluation process).
In the “Transfert de Connaissances en REGion” study (TC-REG project) (40), the elaboration of theory proceeded through two major steps. In step 1, we defined the initial mid-range theory and the knowledge-transfer scheme by i) a literature review of evidence-based strategies for knowledge transfer and mechanisms to enhance evidence-based decision making (e.g., the perceived usefulness of scientific evidence); ii) a qualitative exploratory study in the four regions under study to collect information on the existing actions and resources to implement knowledge-transfer strategies; and iii) a workshop with experts in knowledge transfer, field professionals of the four regions, and TC-REG researchers to choose the strategies to be implemented and develop hypotheses regarding the mechanisms potentially activated by these strategies and any contextual factors potentially influencing them (e.g., the availability of scientific data). In step 2, we validated the initial mid-range theory through two qualitative studies, the first to identify specific actions implemented in the regional knowledge-transfer plan (one series of interviews), and the second to identify the final mid-range theories through two series of interviews and a workshop with the same stakeholders to elaborate the theory by combining strategies, contextual factors, and mechanisms to be activated.
This process is not incompatible with the different stages of experimental evaluations, whatever the steps of their development might be. Indeed, this process could be integrated with experimental designs (15). Figure 3 (Figure 3 – Articulating experimental design and intervention system theory) presents the potential articulation between the intervention system and experimental design based on the three major steps of trials highlighted in the Medical Research Council guideline (3): pilot study, evaluation of effectiveness, and dissemination. The association of a theory-based approach in pilot studies and subsequent effectiveness studies contributes to a better intervention and evaluation design (7), as well as a better understanding of how the intervention works in the designated context. Both contribute to better dissemination of the intervention because of the improvement in its viability under real conditions (36).
For example, we used this process in the OCAPREV study to design an evidence- and theory-based intervention, namely a health application, in a pilot study prior to an evaluation (37). In the EE-TIS study (38), we conducted a randomized control trial with a contribution analysis evaluating a smoking-cessation app (Tabac Info Service). In addition to evaluating outcomes, the study aimed to understand how each component of the app worked for smoking cessation through the mechanisms triggered (e.g. self-efficacy, utility perception, confidence in the app, etc.) and the contextual parameters potentially influencing smoking cessation (e.g., smoking status of the domestic partner; presence of children; others’ support for smoking cessation; family, social, or professional events, etc.).
Moreover, our process is also not incompatible with other theory-driven frameworks such as TOC or realistic evaluation frameworks. For example, according to TOC, interventional components or ingredients are fleshed out and examined separately from the context. The initial hypothesis (the theory of change) is based on empirical or theoretical assumptions. Validation (or its absence), then, addresses the extent to which the explanatory theory, including implementation parameters, corresponds to observations. This makes explicit how inputs, activities, and outcomes are linked and how the various interventional components work together in a causal pathway involving causal inferences and implementation aspects to achieve the impact (10). The difference from our approach is the lack of a focus on mechanisms and on the influence of context; instead, in TOC frameworks, the focus is on the intervention and its specific components. In the systemic approach of an ISyT, contextual elements and mechanisms of effects are actually included in the matrix (5). This is what we did in the OCAPREV study. The theory conceptualized in that study hypothesized 50 causal chains linking behavior sources (capacity, motivation, opportunity to change) with specific behavior-change techniques and mechanisms of effect. Some technical recommendations, i.e., implementation processes or contextual elements, were added to these chains to improve the app’s accessibility, acceptability, and contribution to reducing health inequalities.
In realistic evaluation (33), contextual elements and mechanisms are considered core elements in mid-range theories. Interventional and implementation components, which are included in the ISyT, are not considered. Some authors have proposed including these interventional components, called “resources”, in the definition of mechanisms (39). We do not share this approach, preferring the definition of mechanism suggested by Lacouture et al. (30), who focused on the reaction of stakeholders situated in the context (including their interventional input). Thus, according to our definition, what Dalkin (39) termed resources is an aspect of contextual (pre-existing resources) and interventional components (resources provided by creators and implementers), not part of the triggering mechanisms. Others have distinguished between intervention and context by referring collectively to intervention, context, actors, mechanisms, outcomes (ICAMO) configurations (40). Our interventional system approach does not share this perspective, which blurs the distinction between intervention and context and disregards our position that actors are part of the context. We prefer to keep the tryptic C (Context)–M (Mechanism)–O (Outcome) model by adapting it in this way: Ce (Context external to the intervention)–Ci (interventional context)–M (mechanism)– O (Outcome). For example, in the TC-REG project (40), the aim was to evaluate the conditions of effectiveness of a knowledge-transfer scheme aimed at evidence-based decision making (EIDM) in different public health organizations. The mid-range theories comprised external factors (e.g., initial training of implementers, interest in knowledge-transfer scheme dissemination, leadership profile, political support in the organizations, time to study evidence-based data, team size) (Ce = Context external), interventional components (e.g., access to evidence-based data, training courses, seminars, knowledge brokering, etc.) (Ci = Context interventional), and mechanisms (M) triggered by the combination of both (utility perception of EIDM, motivation to make evidence-based decisions, self-efficacy to analyze and adapt evidence in practice, etc.) to produce outcomes (O).
This process resulted in the creation of eight final mid-range CeCiMO theories about the mechanisms triggered and a final ISyT. This ISyT is presented in Figure 4 below (Figure 4 – ISyT in the TC-REG study).
In this figure, the first level is the outcome: the use of scientific evidence. The second level comprises the eight mechanisms triggered by the knowledge-translation plans implemented, e.g., “professionals perceive them as useful to legitimize or advocate for their professional activity”. The third level shows the knowledge-translation activities (interventional component) triggering the mechanisms, e.g., “communication activities”; “an institutional structure that promotes their use on a daily basis (e.g., dedicated service, transfer plan, and integration into team operations)”; “activities supporting changes in practices (e.g., training, support, and seminars)”. The fourth level represents the contextual elements influencing the activities and also directly influencing the mechanisms, e.g., “political will in favor of KT“; “professionals are aware of the dissemination channels”; “political will and professionals’ experience”; “favorable organizational conditions”. Note that in this TC-REG ISyT, the resources and the details and conditions of implementation of these activities are lacking, but they could be added as Ce and Ci, respectively.