Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

doi:10.21203/rs.3.rs-56645/v3

Download PDF

Research

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

https://doi.org/10.21203/rs.3.rs-56645/v3

This work is licensed under a CC BY 4.0 License

Journal Publication

published 15 Feb, 2021

Read the published version in Implementation Science →

You are reading this latest preprint version

Background: The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

Methods: We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: how is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as grey literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

Results: A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as: 1) the context; 2) a focusing lens; 3) the innovation itself; 4) a lever of influence; 5) an enabler/facilitator or barrier; or 6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

Conclusions: This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Health Policy

Implementation science

Public policy

Evidence-based health care

Systematic review

Critical interpretive synthesis

This study unpacks the implementation of evidence-informed policies and practices through the systematic development of new theory drawing from three distinct fields of scholarship: policy implementation, implementation science and knowledge translation, answering a call from implementation researchers for more integration.
The conceptual framework views implementation from the “outer context” and includes: 1) a model describing the process of implementation and 2) a framework that identifies the policy-related determinants of implementation success.
This conceptual framework, provides researchers, policymakers and implementers with a new way of thinking about implementation and can be used for planning or evaluating implementation efforts.

Implementation has captured the attention of public policy scholars for well over 50 years(1), yet remains relatively under-studied compared to other stages of policy making. The reasons for this are many and include challenges with isolating implementation from other parts of the policy process and a lack of agreement about conceptual underpinnings(2). This then leads to challenges in identifying relevant explanatory variables and analysts often must resort to a “long list of variables that are potentially useful”(2). Even once decisions regarding these challenges have been made, the complex, multi-level and multi-faceted nature of implementation creates difficulties designing and conducting high quality empirical research that can offer useful generalizations to those interested in improving the process of implementation and thus achieving better policy results(2).

Research on implementation has also independently come into sharp focus through the related fields of knowledge translation and implementation science. Conceptual work on implementation from these fields has increased at a seemingly exponential rate to the point where there is a great deal of focus on sorting and classifying the many frameworks, models, and theories and providing guidance toward their use(3–6). The empirical literature is also rapidly increasing, with over 6,200 systematic reviews on consumer-targeted, provider-targeted and organization-targeted implementation strategies in the health field alone (based on a search of www.healthsystemsevidence.org).

Despite the large number of models, theories and frameworks being generated in the knowledge translation and implementation science fields, the role of policy in the implementation process appears to be under-theorized. When policy is included in conceptual work, it is often identified as a contextual variable(7,8) rather than being central to the implementation concept itself. It is also often presented as a broad category of “policy”, rather than as a variable that is specific and therefore measurable in empirical work. This lack of conceptual clarity and empirical work about policy and other policy-related structural constructs has been noted by several researchers. For example, a systematic review of measures assessing the constructs affecting implementation of health innovations makes specific reference to the “relatively few” measures available to assess structural constructs, which they define as “political norms, policies and relative resources/socioeconomic status”(9). As a result, the field of public policy appears to have on the one hand, a challenge of too many policy-related implementation variables, and on the other hand, the fields of knowledge translation and implementation appear to have too few.

In recent years some researchers have recognized these silos in scholarship and have called for more implementation research that integrates public policy and implementation science and knowledge translation perspectives(10). For example, Johansson concludes that implementation problems could be better understood through the inclusion of research in public administration, with more focus on issues such as resource allocation, priorities, ethical considerations, the distribution of power between actors and organizational boundaries(11).

In addition to these challenges, much of the seminal policy scholarship on implementation from both the public policy and knowledge translation and implementation literatures come from the United States(12–15). This has resulted in a concentration of theoretical and empirical works that reflect the governance, financial and delivery arrangements that are particular to the US(16,17) and that may not always readily apply in other contexts. These differences are particularly marked when it comes to the policy domain of health given the differences of the US system compared to most others(18). One notable exception to this is the European contributions to the “second generation” of policy scholarship on implementation, which adopted the perspective of those at the “coal face” of policy implementation(19).

In response to these challenges, the objective of our study was to develop an integrated theoretical framework of the implementation process from a policy perspective by combining findings from the public policy, implementation science and knowledge translation fields. By integrating knowledge from these fields using a critical interpretive synthesis approach, we specifically examine how policy considerations are described in implementation theories, frameworks and processes from existing published and grey literature. Our goal was to generate a theoretical framework to foster an improved understanding of the policy contributions to implementation that can be used in future studies to generate testable hypotheses about large-scale system implementation efforts.

Study design

Given the broad goal of this study, the question of interest and the scope of potentially applicable literature from discrete fields that could inform this work, we selected a critical interpretive synthesis (CIS) approach. Drawing from the techniques of meta-ethnography combined with traditional systematic review processes, CIS employs an inductive and interpretive technique to critically inspect the literature and develop a new conceptualization of the phenomenon of interest. Unlike traditional systematic reviews that often focus on questions of effectiveness, CIS is helpful in generating midrange theories with strong explanatory power (20,21). This is suitable for our goal of developing a conceptual framework that better integrates findings from diverse fields and affords the opportunity to critically inspect both individual studies and the literature from each field as a whole in terms of the nature of the assumptions underlying each field, and what has influenced their proposed solution(22). The method begins with a compass question, which evolves throughout the course of the review(22,23). Our compass question was: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success?

Review scope

Our review casts a very broad net in terms of implementation processes and theories. While our main focus is on large-scale implementation efforts in health, behavioural health and human services areas that are not specific to a particular condition, we also drew from other large-scale implementation theories and empirical work, such as from the field of environmental science, that may yield important insights toward a more integrated framework of implementation. We drew from two key sources of literature: 1) existing frameworks, models, and theories (public policy, implementation science and knowledge translation); and 2) empirical studies that report on specific implementation processes.

Given our interest in implementation processes from a policy perspective, we limited our review to implementation frameworks, models, theories and empirical reports that describe implementation efforts at a community or systems level (e.g., city, province/state or country) where policy considerations are most likely to be an important factor. Implementation of a single evidence-based practice (unless across a large-scale) or implementation in a single organization were excluded, as was research that focused on behavior change at the individual level.

Electronic search strategy

Using the compass question, and in consultation with a librarian, we constructed a table of Boolean-linked key words and then tested several search strategies (Table 1). The search was then conducted in October 2020 for the time period of January 2000 – September 2020 using the following 12 databases: ASSIA, CINAHL (via EBSCO), EMBASE (via Ovid), ERIC, Health Star (via Ovid), MEDLINE (via Ovid), PAIS Index, PolSci, PsychINFO, Social Sciences Abstracts, Social Services Abstracts and Web of Science. The dates for the policy databases (PolSci and Social Sciences Abstracts) were extended to 1973 to ensure key conceptual articles would be retrieved, such as the seminal work by Sabatier and Mazmanian in 1980(14). A grey literature search was also conducted using Health Systems Evidence (which indexes policy documents related to health system arrangements and implementation strategies, as well as systematic reviews). Similar search strings were used across all databases with minor adjustments to ensure searches were optimized. We prioritized sensitivity (comprehensiveness) over specificity (precision) in our search strategy.

Article selection

We excluded articles based on their titles and abstracts if they did not fit within the study scope or if they were not conceptual or empirical works. We created additional inclusion criteria that were based on the following questions: 1) Is there a moderate (or greater) chance that the article will shed light on the role of policy in an implementation process or on the outcomes of the process? 2) Does the article describe implementation efforts at a community or systems level? and 3) Does the article identify actors at the government, organizational or practice level such as policy entrepreneurs who may be central to policy implementation efforts? Any articles that did not meet at least one of these criteria were excluded.

Complementary to the formal search, and in keeping with the inductive strategies that are part of the CIS process, we also conducted hand searches of the reference lists of relevant publications and searched the authors’ personal files to identify further articles and theoretically sampled additional articles to fill conceptual gaps as the analysis proceeded.

After completing the searches, an Endnote database was created to store and manage results. Once duplicates were removed, a random selection of two percent of the articles was independently screened by two reviewers (H.B. & A.M.) who were blinded to each other’s ratings and used the same inclusion criteria. The reviewers classified each title and abstract as “include” “exclude” or “uncertain”. Inter-rater agreement was determined using the kappa statistic. This process was undertaken to improve the methodological rigor by enhancing trustworthiness and stimulating reflexivity, not to establish a quantitative assessment per se(24). Any discrepancies were then discussed between reviewers until consensus was reached. Next, one reviewer assessed the remaining titles and abstracts. Articles classified as “include” or “uncertain” were kept for full-text review.

The full text of the remaining articles was then assessed by one reviewer. Articles were excluded at this stage if they did not provide detailed insight into the compass question. Articles were also sorted according to whether they were a conceptual contribution (i.e., presented a model, theory, framework or theoretical concept on implementation) or an empirical contribution (i.e., used qualitative, quantitative, review or other research methods to present new findings or an analysis of implementation).

Data analysis & synthesis

Our data analysis proceeded in four stages. First, while screening and assessing the articles for inclusion, we noted some general observations of how policy was incorporated in the literature from each field of interest (policy/public administration, implementation and knowledge translation). Second, we classified articles according to how policy was portrayed in implementation theory and processes. Third, we constructed a data extraction template for conceptual and empirical studies that included: 1) descriptive categories (the author(s), the name of the model, theory or framework (if provided), year of publication, author location, focus of the article and whether a graphic or visual aid was included); 2) content from the article that addressed the compass question regarding how policy is portrayed and what aspects are important for success; and 3) interpretive categories including “synthetic constructs” developed by the review team from the article and additional notes on how the article contributed to the development of the conceptual model. Additionally, the data extraction form for the conceptual articles included a classification of the type of framework according to Nilsen’s taxonomy of implementation models, theories and frameworks(3).

In the fourth and final stage, we initially focused on the conceptual literature and used it as a base from which to build our integrated conceptual model. We developed the synthetic constructs by reviewing the content from each article that addressed the compass question and interpreting the underlying evidence using a constant comparative method to ensure that the emerging synthetic constructs were grounded in the data, similar to a grounded theory approach (25). These synthetic constructs were then used to begin to build the conceptual model and an accompanying graphic representation of it. We then critiqued the emerging constructs to identify gaps in the evidence and emerging constructs.

Using this emerging model, we purposively sampled additional conceptual literature to fill the gaps that we identified and to ensure we incorporated as many relevant concepts as possible. We did this by consulting reviews of existing models, theories and frameworks(2–6) to identify additional relevant concepts not captured by our search strategy and by hand searching the reference sections of some seminal conceptual papers(7,26). Once saturation of the conceptual literature was reached, we purposively sampled a subset of the empirical literature and used this subset to “test” the model and add additional detail to the theoretical constructs gleaned from empirical report. We used a similar data extraction template with the exceptions of removing the descriptive category of model or theory name and the interpretive classification using the Nilsen taxonomy (3), but adding the descriptive category of “methodology”. If our model did not capture findings from the empirical studies, we revised it and re-tested. This process continued until saturation was reached and additional empirical studies yielded no further insights into our model.

The methods reported here are based on a protocol developed prior to initiating the study. The protocol and a note about the four ways that the reported methods differed from the protocol are available upon request.

Search results and article selection

Our database search retrieved 16,107 documents and 11,434 unique documents once duplicates were removed. The review of titles and abstracts was completed independently by two reviewers on a random sample (n = 171) of the documents. The Kappa score was 0.72 indicating substantial agreement. Figure 1 provides a flow diagram outlining the search strategy. Following these criteria for the remaining titles and abstracts resulted in 1,208 documents included for full text review. The full text review excluded an additional 940 documents leaving 268 potentially relevant documents (excluded documents and the rationale for exclusion are available upon request). Of these, 23 conceptual documents, 243 empirical documents and two documents that included both conceptual and empirical elements were included for the data extraction and analysis phase. We sampled and extracted data on all of the conceptual articles. For the empirical articles, we chose a maximum variation sampling approach based on the subject matter and article topic with an initial sample of 10% of the articles. We also noted that nine of the articles related to a large, multi-year national implementation study(27–35). Because this was the largest and most comprehensive account of the role of policy in large-scale implementation efforts identified through our search, we included these as a sub-group for data extraction. This approach led to data extraction for 34 empirical articles.

In addition to these two approaches we sampled articles that filled in conceptual gaps as our model developed. This process resulted in the retrieval of an additional 26 conceptual articles and 3 empirical articles. In total, 86 unique documents were included with two of these documents used in both the conceptual and empirical data extraction (Tables 2a and 2b). While our process was inclusive of English language publications from any country, the majority of articles were conducted by US researchers (n = 57), with the others coming mainly from other Western countries (the United Kingdom (n = 8), Netherlands (n = 7), Australia (n = 5), Canada (n = 2), Sweden (n = 2), Germany (n = 2), and Europe, China and OECD (n = 1)). Articles covered a range of topics including health and health care, public health, mental health and addictions, children and youth, social care, justice, and climate change, among others. The conceptual documents included all of the categories of theories, models and frameworks identified by Nilsen (3), with the Determinants Framework type being most common. The empirical articles employed a wide array of methods that fall into the broad categories of qualitative, quantitative and mixed methods.

General observations

Through this process we noted several general observations regarding the characteristics of existing literature. In terms of the scholarly disciplines, most of the implementation science literature focused on the organizational or service provider levels with an emphasis on changing practice, often by introducing an Evidence-Informed Policy or Practice (EIPP). The knowledge translation literature included policymakers as a target audience for research evidence, but the focus was on the agenda setting or policy formulation stages of the policy cycle, as opposed to the implementation of an EIPP. Here, the scholarship focused on strategies to increase the use of evidence in policy decision-making. The public policy literature included theory describing “top-down”, “bottom-up” and integrated approaches to implementing an EIPP. The object of implementation in this area was the policy itself, rather than a specific program or practice. There was often no clear articulation of independent and dependent policy-related implementation variables across any field, although many articles did partially address this.

How policy is described in implementation theory and processes

Our coding based on the compass question resulted in the following characterization of how policy is described in implementation theory and processes:

Policy is described as:

Context in which implementation occurs (i.e., only briefly citing a policy as the reason for implementation)
Focusing lens, signaling to systems what the priorities should be (i.e., referring to policy statements or attention by policymakers as a signal about what is important to prioritize)
Innovation itself – the implementation object (i.e., the “thing” being implemented is policy, such as new legislative policy on tobacco cessation).
Lever of influence in the implementation process (i.e., policy is identified as at least one of the factors influencing the implementation process)
Enabler/facilitator or barrier to implementation (moderating variable) (i.e., while policy is identified as being external to the implementation effort, it is later found to be a barrier or facilitator to implementation)
Outcome – the success of the implementation process is at least partially defined and measured by a change in policy
Policy actors as important participants or leaders in implementation

Theoretical framework

Our approach to developing the theoretical framework was two-fold. The findings from our analysis suggested constructs that addressed both the process of implementation and the factors underpinning the success or failure of implementation. We therefore first developed a process model(3) that describes the steps in the process of translating EIPPs into effectively embedded system changes. Next, we developed a determinants framework, which specifies the types of policy determinants (independent variables) that affect implementation outcomes (dependent variables). This two-part theoretical framework achieves two goals: 1) the process model is most useful in describing the process of implementation from a policy perspective, and 2) the determinants framework is most useful for understanding and explaining policy-related influences on implementation outcomes.

Part 1 – Process Model

Figure 2 depicts this novel process model focusing on one policy or system level. What follows is a narrative description of the model.

Policy is shaped as it moves through systems. The process through which policy travels from one level to another is known as policy transfer(36–38). Each policy level is nested in a context that includes existing ideas (values, evidence, etc.), interests (interest groups, civil society, etc.), institutions (existing rules and institutional structures) and external factors (natural disaster, change in economic conditions) that affect the interpretation of the policy package(39,40). This context affects how a problem is defined, whether it has the attention of decision makers and whether it is up for active decision-making. This aligns with the “problem definition” and “agenda setting” stages of the policy cycle but is also described as part of the “exploration phase” in implementation science(12,41). Once a decision has been reached that something should be done to address a given issue, attention shifts to the “policy development” stage of the policy cycle, which aligns with the “adoption decision and preparation” stage of implementation. It is during the policy development/adoption decision and preparation stage that the policy package gets developed.

Policy package

A policy package usually includes a mix of policy levers or instruments, including legal and regulatory instruments, economic instruments, voluntary instruments, or information and education instruments(42,43). The policy package can also include some implementation guidance such as a description of the overall implementation strategy architecture, the major streams of activity, timing of events and milestones, and, roles and responsibilities.

The level of ambiguity of the policy package in terms of its goals and means of attaining them, and the amount of conflict among actors with respect to the policy package are important to help characterize the implementation process and to explain its outcomes. According to Matland (44) the consideration of ambiguity and conflict leads to four types of implementation processes: 1) Administrative implementation occurs when there is low policy ambiguity and low policy conflict (e.g., eradication of small pox); 2) Political implementation occurs when there is low ambiguity but high levels of conflict (e.g., public transit); 3) Experimental implementation occurs when there is high ambiguity but low conflict (e.g., Head Start programs for young children); and 4) Symbolic implementation occurs when both ambiguity and conflict are high and policies only have a referential goal and differing perspectives on how to translate the abstract goal into instrumental actions (e.g., establishing youth employment agencies).

Implementation process

The policy implementation process can start at any level, move in any direction and can “skip” levels. Power also shifts as implementation proceeds through levels(29,45). The level with the most implementation activity tends to have the most power. This is true not only for different levels of governance, but as implementation cascades across organizations, through “street level bureaucrats”(13) and on to the end-user or target population (the “recipient”) of the implementation process. Policy decisions at one level becomes context for other levels. Implementation activities at one level can exert either direct or indirect effects on another level. The context surrounding each level (prevailing ideas, interests, institutions and external events) influences the acceptability and ultimate success of implementation. Finally, the overall implementation approach may need to shift over time in response to a constantly evolving context. For example, one study found it necessary to change the implementation approach for a road safety program in respond to changes in policy authority(46).

Outcomes

The process of implementation is undertaken in order to lead to outcomes, which can be separated and measured at different levels. Proctor and colleagues(47) identifies three separate outcomes: 1) implementation outcomes; 2) service outcomes; and 3) recipient-related outcomes. Along with these outcomes, our model includes policy- and systems-level outcomes. These can be evaluated according to the policy outputs (i.e., enforcement variables, change of perspective of street-level staff, etc.), policy outcomes (i.e., unemployment levels, life-expectancy of population, etc.) or indices of policy system change (i.e., administrative re-organization, privatization, etc.)(45). While the measures and levels will vary depending on the size, scale and focus of implementation, there is broad agreement that outcomes should be clearly defined a-priori and precisely measured. Evaluation findings regarding outputs and outcomes can dynamically feed back into the implementation process as it unfolds. This creates feedback loops and the process becomes very dynamic and multi-directional.

Part 2 – Determinants framework

Figure 3 presents an overview of our determinants framework and the relationship among the determinants. Our findings point to three sets of policy-related factors that affect the process, outputs and outcomes of implementation: 1) policy instruments and strategies; 2) determinants of implementation; and 3) policy actors, including their characteristics, relationships and context. Collectively, these feed into the process of implementation that proceeds in an iterative fashion along the stages: exploration, installation/preparation, initial implementation, full implementation/sustainment(12,41). The types of policy influences vary according to the stage of implementation(12). The process of implementation leads to a variety of outputs and outcomes as described above.

Policy instruments and strategies

Policy instruments and strategies are the most common set of factors mentioned in the literature and we found evidence for each of the instrument types described here, although with varying levels of detail. Policy instruments can be applied to implementation in differing ways, often with two or three levers used concurrently to implement a single initiative or strategy(48). In order to classify these strategies in a meaningful way, we drew on and adapted elements of a mutually exclusive and collectively exhaustive framework that identifies key features of health and social systems(39) and honed in on strategies that are particularly important for implementation (Table 3). These include strategies focused on the governance arrangements, financial arrangements, service delivery arrangements and implementation-related supports in systems. We then divided these strategies according to the intended “target” of implementation. Common targets of implementation from a policy perspective include the whole system, organizations, the workforce or service providers, consumers, and the innovation itself (the EIPP to be implemented). We wish to note, however, that because policy-related variables have not necessarily been treated with the same specificity as other types of implementation variables, the most common strategies do not reflect the full array of strategies that could be employed.

Determinants

Our framework identifies eight categories of determinants (see “Determinants” box and elsewhere in Figure 3). Each of these categories represents a suite of factors that are hypothesized to independently affect implementation outcomes. These determinants are described briefly below and in more detail in Table 4.

I – Characteristics of the Evidence-Informed Policy or Practice (EIPP) - The success or failure of a particular policy package cannot be evaluated based on its intrinsic characteristics alone (45). Instead it is important to examine whether the policy selected is an appropriate “fit” with the problem(49), well-justified(50) and aligned with existing context(12,51).

II - Policy Formulation Process - This is the shape given to a policy by the initial formation processes(45). It includes who in government is responsible for formulating the policy, their legitimacy and the extent to which there is opportunity to provide feedback, how much feedback is given and the responsiveness in terms of adjustments made(45).

III – Vertical Public Administration and Thickness of Hierarchy - Vertical public administration is the term used to identify the layers in the policy transfer process. It refers to separate governments exercising their authority with relative autonomy(45). Policies generated outside of a socio-political level may be more or less acceptable to that level. Within a given layer, a particular policy area may require the mobilization of any number of institutions, departments or agencies and these agencies must act in a coordinated, interdependent fashion, termed “thickness of the hierarchy”(52).

IV – Networks/Inter-organizational Relationships - The existence and nature of the relationships between parallel organizations who must collaborative in order to achieve effective implementation and who do not have a hierarchical relationship(45).

V – Implementing Agency Responses - The factors affecting the responses of implementing agencies can be divided into issues related to the overall characteristics of the agencies and the behaviour of front-line or street-level staff(13,45).

VI – Attributes and Responses from Those Affected by EIPP- Attributes include the diversity of target group behaviour and the target group as a percentage of the population(14). Responses include thing like impacts on workforce stability(12).

VII – Timing/Sequencing - As implementation is a process that unfolds over time, it does not always align with the cycles to which it is subject and the time constraints inherent therein(53,54). Additionally, the external context in which implementation occurs is ever changing and “quintessentially unstable”, and success hinges on the ability to perceive those changes and take the necessary actions to adjust along the way(55). In Figure 3, Timing/Sequencing is placed outside of the Determinants box to reflect its importance across all of the other elements.

VIII – External Environment or Policy Context – Much of the literature identified factors outside the policy area of focus that may influence implementation (Figure 3, outside the hatched line). Many authors referred to this generally as the “political and social climate”, as unmodifiable or macro “context” or as “socio-economic conditions”(9,14,56–61). We organized this determinant using: 1) the 3I+E framework(62), and 2) a taxonomy of health and social system arrangements(63).

In general, these categories of determinants should be viewed as interactive and not completely discrete(45) and the inter-relationship among the determinants is key(37).

Policy actors

Our analysis revealed a wide range of policy actors who are important for implementation. In an attempt to create a category of variables that is analytically useful across contexts, we first divided the types of policy actors into the broad categories of: political actors, bureaucratic actors, special interests and experts(64). To provide more specificity, we further divided these into a non-exhaustive list of actor sub-types that were frequently mentioned in the literature and included examples of the types of roles they tend to assume in implementation (Table 5). While many of the sub-types are commonly identified in other phases of the policy cycle, some receive particular attention in the implementation literature. These include two types of Special Interests: 1) Implementing Agencies - organizations or programs that are responsible for implementing the EIPP (e.g., hospitals, schools, etc.); and 2) Street-Level Bureaucrats who, due to the relatively high degree of discretion in their jobs, and therefore discretion over the dispensation of public benefits or sanctions to citizens, can be critical to realizing any large-scale implementation efforts. There are also three Expert sub-types that are particularly visible during implementation: 1) Field or Practice Leaders who can be influential in supporting practice change amongst professionals; 2) Innovation Developers/Disseminators who have developed the EIPP to be implemented and who may contribute or adapt tools and other types of support to encourage successful implementation; and 3) Intermediaries/Technical Assistance Providers who are organizations, programs or individuals that work “in between” policymakers, funders, and frontline implementers, to facilitate effective implementation drawing on expertise in implementation.

There are also three categories of actor-related variables that are important: 1) actor characteristics; 2) actor relationships; and 3) the context in which the actors are embedded. (Figure 4). First, the characteristics of the policy actors (either individual- or organizational-level) such as their knowledge of the implementation context, their legitimacy, power and control, and their leadership in the context of the implementation effort are often cited as being critical to the success in large-scale implementation initiatives. Second, the relationships policy actors have with other actors, such as the level of shared values and beliefs or the coordination and alignment of actors and their activities, can be predictive of successful implementation. Finally, the context of the actors, such as the sustainment of political will and commitment and the stability of the actors themselves can predict the long-term success of implementation.

Our study represents one of the first comprehensive attempts to answer the call of scholars to integrate the fields of implementation science, knowledge translation and policy implementation in an effort to build a more comprehensive and accurate understanding of implementation. By integrating conceptual and empirical works from all three fields, the resultant two-part theoretical framework provides additional clarity regarding the process of implementation viewed from a policy perspective and identifies a number of policy-related determinants that can be tested empirically in the future.

A key strength of our study was the methodological approach we took to theory building.

First was the comprehensiveness of the search strategy, which aimed to identify scholarship from more than one academic discipline and across wide range of topics beyond health. The literature identified through the search process revealed some interesting parallels and unique differences between the fields that made it clear to us the extent of the lack of integration up to this point. Perhaps not surprisingly, the area of public health seemed to be the most fertile ground for integration. This is likely due to their focus on population-level concerns requiring system-wide implementation of EIPPs and a diverse implementation ecosystem. The search strategy was part of the mixed methods approach of the CIS, which blended the rigor of a systematic search methodology that is explicit and replicable, with the inductive, iterative and purposive sampling techniques from qualitative review methods to build mid-range theory. The result is a theoretical framework that is clearly linked to the literature, which should instill some confidence in the academic community regarding its grounding. Critical interpretive synthesis is a relatively new approach but is growing in popularity for these reasons.

Despite the merits of our approach, we did identify some challenges. First, we believe the literature from public policy may be underrepresented for several reasons: 1) search terms did not retrieve as much from those fields (it could be that there are terms used more commonly in those fields that would have increased yield); 2) the disciplinary approach to the scholarship in public policy often means the articles were less explicit about methods and this meant that more were excluded as not being “high yield”; 3) more of that scholarship is captured through other media (e.g., books) and while some of these were included, our approach was not as sensitive to retrieving these types of documents. We also did not include all of the empirical articles for data extraction and we may have missed a key theme or framework component. While we believe this is unlikely because we continued to sample until saturation was reached, it is still possible something was missed. Finally, there were few documents from low- and middle-income countries included in the final sample. Specific efforts to include relevant documents from LMICs in future may enrich and refine the model.

As a result of this research, policymakers and practitioners looking to use a conceptual model to guide their implementation activities have two additional options that they can be confident draw from existing theory and empirical works. Large-scale implementation endeavors or those that have started small and are looking to scale-up should at least be mindful of the critical roles of policy during the process and what policy-related factors may be important for success. Those planning implementation activities can consider the elements presented in the framework as factors that may require consideration and adjustment prior to implementing something new. Our work supports thinking beyond the program or practice levels and unpacks policy considerations that may have influence on, or affect the effectiveness of, a program or practice. Furthermore, the inclusion of policy-related outputs and outcomes in our framework offers policymakers and practitioners the option of additional indicators of success on which they can measure and report.

Like any new theoretical contribution, our framework would benefit from further refinement and testing by the research community. Future research could adopt the process model to guide a policy-intensive implementation effort and test it to determine its usefulness in such efforts. Researchers could also select particular framework elements and unpack them further for additional precision and clarity, drawing from multiple fields of scholarship. Our framework also offers some much-needed policy variables that have been lacking in the implementation science and knowledge translation fields, which could be incorporated as part of a suite of variables in implementation research.

Our study represents an early effort at integrating the fields of public policy, implementation science and knowledge translation. We have learned that there is indeed a great deal that each of the fields can learn from the other to advance our understanding of policy- and systems-level implementation efforts and hope that these efforts are followed by more interdisciplinary research in order to truly bridge this divide.

Ethics approval and consent to participate:

Not applicable

Consent for publication:

Not applicable

Availability of data and materials:

Not applicable

Competing interests:

The authors declare they have no competing interests.

Funding:

This study was partially supported through a doctoral scholarship from the P.E. Trudeau Foundation.

Authors’ contributions:

HLB was responsible for conceiving of the focus and design of the study (with support from JNL) and for completing all data collection, analysis and interpretation. JNL also contributed to the analysis during ongoing iterative cycles of interpretation and synthesis that led to the development of the final theoretical model. AM independently assessed a sub-sample of the documents for eligibility and worked with me to refine the inclusion criteria. HLB drafted the manuscript, and JNL, MGW, and GM provided comments and suggestions that were incorporated into revisions. All of authors approved the final version of the manuscript.

Acknowledgements:

Not applicable

Pressman JL, Wildavsky AB. Implementation. How Great Expectations in Washington Are Dashed in Oakland. Berkeley, CA: University of California Press; 1973.
Hill M, Hupe P. Implementing public policy: Governance in theory and in practice. 3 ed. London, UK: SAGE; 2014.
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.
Moullin JC, Sabater-Hernandez D, Fernandez-Llimos F, Benrimoj SI. A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework. Heal Res Policy Syst. 2015;13:16.
Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.
Mitchell SA, Fisher CA, Hastings CE, Silverman LB, Wallen GR. A thematic analysis of theoretical models for translational science in nursing: mapping the field. Nurs Outlook. 2010;58(6):287–300.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. Am J Community Psychol. 2012;50(3–4):445–59.
Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.
Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet?--a comparison of implementation science and policy implementation research. Implement Sci. 2013;8:63.
Johansson S. Implementing evidence-based practices and programmes in the human services: lessons from research in public administration. Eur J Soc Work. 2010;13(1):109–25.
Aarons GA, Hurlburt M, Horwitz SMC. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Heal Ment Heal Serv Res. 2011 Jan;38(1):4–23.
Lipsky M. Street-level bureaucracy: Dilemmas of the individual in public services. New York: Russell Sage Foundation; 1980.
Sabatier P, Mazmanian D. The implementation of public policy: A framework of analysis. Policy Stud J. 1980;8(4):538–60.
Van Meter DS, Van Horn CE. The Policy Implementation Process: A Conceptual Framework. Adm Soc. 1975;6(4):445–88.
O’Toole Jr LJ. Research on policy implementation: Assessment and prospects. J public Adm Res theory. 2000;10(2):263–88.
Saetren H. Implementing the third generation research paradigm in policy implementation research: An empirical assessment. Public Policy Adm. 2014;29(2):84–105.
Papanicolas I, Woskie LR, Jha AK. Health care spending in the United States and other high-income countries. JAMA - J Am Med Assoc. 2018;319(10):1024–39.
Hjern B. Implementation research—the link gone missing. J Public Policy. 1982;2(3):301–8.
Flemming K. Synthesis of quantitative and qualitative research: an example using Critical Interpretive Synthesis. J Adv Nurs. 2010;66(1):201–17.
Wilson MG, Ellen ME, Lavis JN, Grimshaw JM, Moat KA, Shemer J, et al. Processes, contexts, and rationale for disinvestment: a protocol for a critical interpretive synthesis. Syst Rev. 2014;3(1):143.
Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6(1):35.
Entwistle V, Firnigl D, Ryan M, Francis J, Kinghorn P. Which experiences of health care delivery matter to service users and why? A critical interpretive synthesis and conceptual map. J Health Serv Res Policy. 2012;17(2):70–8.
Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Serv Res. 1999;34(5 Pt 2):1189.
Charmaz K. Constructing Grounded Theory: Second Edition. 2nd ed. Constructing Grounded Theory. Sage Publications; 2014.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly 2004 p. 581–629.
Bond GR, Drake RE, McHugo GJ, Rapp CA, Whitley R. Strategies for improving fidelity in the National Evidence-Based Practices Project. Res Soc Work Pract. 2009;19(5):569–81.
Finnerty MT, Rapp CA, Bond GR, Lynde DW, Ganju V, Goldman HH. The State Health Authority Yardstick (SHAY). Community Ment Heal J. 2009;45(3):228–36.
Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco J, et al. The state policy context of implementation issues for evidence-based practices in mental health. Psychiatr Serv. 2007;58(7):914–21.
Isett KR, Burnam MA, Coleman-Beattie B, Hyde PS, Morrissey JP, Magnabosco JL, et al. The role of state mental health authorities in managing change for the implementation of evidence-based practices. Community Ment Heal J. 2008;44(3):195–211.
Jones AM, Bond GR, Peterson AE, Drake RE, McHugo GJ, Williams JR. Role of state mental health leaders in supporting evidence-based practices over time. J Behav Heal Serv Res. 2014;41(3):347–55.
Mancini AD, Moser LL, Whitley R, McHugo GJ, Bond GR, Finnerty MT, et al. Assertive community treatment: facilitators and barriers to implementation in routine mental health settings. Psychiatr Serv. 2009;60(2):189–95.
Peterson AE, Bond GR, Drake RE, McHugo GJ, Jones AM, Williams JR. Predicting the long-term sustainability of evidence-based practices in mental health care: an 8-year longitudinal analysis. J Behav Heal Serv Res. 2014;41(3):337–46.
Rapp CA, Bond GR., Becker DR., Carpinello SE., Nikkel RE., Gintoli G. The role of state mental health authorities in promoting improved client outcomes through evidence-based practice. Community Ment Health J. 2005;41(3):347–63.
Rapp CA, Goscha RJ, Carlson LS. Evidence-based practice implementation in Kansas. Community Ment Health J. 2010;46(5):461–5.
Dolowitz DP, Marsh D. Learning from abroad: The role of policy transfer in contemporary policy‐making. Governance. 2000;13(1):5–23.
Evans M, Davies J. Understanding Policy Transfer: A Multi-Level, Multi-Disciplinary Perspective. Public Adm. 1999;77(2):361–85.
Bauman AE, Nelson DE, Pratt M, Matsudo V, Schoeppe S. Dissemination of physical activity evidence, programs, policies, and surveillance in the international public health arena. Am J Prev Med. 2006;31(4 Suppl):S57-65.
Lavis JN, Rottingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9(3):e1001186.
Shearer JC, Abelson J, Kouyate B, Lavis JN, Walt G. Why do policies change? Institutions, interests, ideas and networks in three cases of policy reform. Heal Policy Plan. 2016;31(9):1200–11.
Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F, Network TNIR. Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute; 2005.
Howlett M. Beyond good and evil in policy implementation: Instrument mixes, implementation styles, and second generation theories of policy instrument choice. Policy Soc. 2004;23(2):1–17.
Treasury Board of Canada Secretariat. Assessing, selecting, and implementing instruments for government action. Ottawa: Governement of Canada; 2007.
Matland RE. Synthesizing the implementation literature: The ambiguity-conflict model of policy implementation. J Public Adm Res Theory. 1995 Apr;5(2):145–74.
Hill M, Hupe P. Implementing public policy: Governance in theory and in practice. 1st ed. London, UK: Sage; 2002.
Bax C, de Jong M, Koppenjan J. Implementing evidence-based policy in a network setting: road safety policy in the Netherlands. Public Adm. 2010;88(3):871–84.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Heal. 2011;38(2):65–76.
Grace FC, Meurk CS, Head BW, Hall WD, Carstensen G, Harris MG, et al. An analysis of policy levers used to implement mental health reform in Australia 1992-2012. BMC Health Serv Res. 2015 Oct 24;15(1):479.
Grundy J, Smith M. Evidence and equity: Struggles over federal employment equity policy in Canada, 1984-95. Can Public Adm publique du Canada. 2011 Sep;54(3):335–57.
Viennet R, Pont B. Education Policy Implementation. OECD Education Working Papers Series. 2017.
Fleuren MAH, Paulussen TGWM, Dommelen P, Van Buuren S. Towards a measurement instrument for determinants of innovations. Int J Qual Heal Care. 2014 Oct 1;26(5):501–10.
Hill M, Hupe P. The multi-layer problem in implementation research. Public Manag Rev. 2003;5(4):471–90.
Culotta D, Wiek A, Forrest N. Selecting and coordinating local and regional climate change interventions. Environ Plan C-Government Policy. 2016;34(7):1241–66.
Evans BA, Snooks H, Howson H, Davies M. How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implement Sci. 2013;8.
Pettigrew A, Whipp R. Managing change and corporate performance. In: European industrial restructuring in the 1990s. Springer; 1992. p. 227–65.
Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: A framework for action. PLoS Med. 2005;2(7):600–5.
Burris S, Mays GP, Scutchfield FD, Ibrahim JK. Moving from intersection to integration: Public health law research and public health systems and services research. Vol. 90, Milbank Quarterly. 2012. p. 375–408.
Harris JR, Cheadle A, Hannon PA, Lichiello P, Forehand M, Mahoney E, et al. A framework for disseminating evidence-based health promotion practices. Prev Chronic Dis. 2012 Jan;9(1).
Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci. 2008;3:26.
Strehlenert H, Richter-Sundberg L, Nyström ME, Hasson H, Nystrom ME, Hasson H. Evidence-informed policy formulation and implementation: a comparative case study of two national policies for improving health and social care in Sweden. Implement Sci. 2015 Dec 8;10(1).
Wisdom JP, Chor KHB, Hoagwood KE, Horwitz SM. Innovation adoption: A review of theories and constructs. Adm Policy Ment Heal Ment Heal Serv Res. 2014;41(4):480–502.
Lavis JN. Studying health-care reforms. In: Lazar H, Lavis J, Forest P-G, Church J, editors. Paradigm freeze: why it is so hard to reform health care in Canada. Kingston: McGill-Queen’s University Press; 2013.
Lavis JN, Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, et al. Developing and refining the methods for a “one-stop shop” for research evidence about health systems. Heal Res Policy Syst. 2015 Feb 25;13(1):10.
Dente B. Who Decides? Actors and Their Resources. In: Understanding Policy Decisions. Springer; 2014. p. 29–66.
Bowen SAK, Saunders RP, Richter DL, Hussey J, Elder K, Lindley L. Assessing levels of adaptation during implementation of evidence-based interventions: introducing the Rogers-Rutten framework. Heal Educ Behav. 2010;37(6):815–30.
Bruns EJ, Hoagwood KE, Rivard JC, Wotring J, Marsenich L, Carter B. State implementation of evidence-based practice for youths, part II: Recommendations for research and policy. J Am Acad Child Adolesc Psychiatry. 2008;47(5):499–504.
Campos PA, Reich MR. Political Analysis for Health Policy Implementation. Heal Syst Reform. 2019;5(3):224–35.
Cherney A, Head B. Supporting the knowledge-to-action process: a systems-thinking approach. Evid Policy. 2011;7(4):471–88.
Chin MH, Goldmann D. Meaningful disparities reduction through research and translation programs. JAMA. 2011;305(4):404–5.
Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Adv Sch Ment Health Promot. 2008;1(3).
Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43.
Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within health care organizations. Literature review and Delphi study. International Journal for Quality in Health Care. 2004.
Godfrey JL. Re-implementing Assertive Community Treatment: One agency’s challenge of meeting state standards. Diss Abstr Int Sect B Sci Eng. 2011;72(4-B):2434.
Green LW, Orleans CT, Ottoson JM, Cameron R, Pierce JP, Bettinghaus EP. Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control. Am J Prev Med. 2006;31(4 Suppl):S66-81.
Greig G, Entwistle VA, Beech N. Addressing complex healthcare problems in diverse settings: Insights from activity theory. Soc Sci Med. 2012;74(3):305–12.
Harris J, Cheadle A, Hannon P, Forehand M, Lichiello P, Mahoney E, et al. A Framework for Disseminating Evidence-Based Health Promotion Practices. Prev Chronic Dis. 2011 Dec;
Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016 Mar 10;11(1):33.
Hendriks A, Jansen M, Gubbel J, De Vries N, Paulussen T, Kremers S. Proposing a conceptual framework for integrated local public health policy, applied to childhood obesity--the behavior change ball. Implement Sci. 2013 Apr 18;8(1):46.
Hodges S, Ferreira K. A multilevel framework for local policy development and implementation. In: Child and family advocacy: Bridging the gaps between research, practice, and policy. New York, NY: Springer Science; 2013. p. 205–15.
Hupe PL. The thesis of incongruent implementation: Revisiting Pressman and Wildavsky. Public Policy Adm. 2011;26(1):63–80.
Hupe PL, Hill MJ. ‘And the rest is implementation.’Comparing approaches to what happens in policy processes beyond Great Expectations. Public Policy Adm. 2016;31(2):103–21.
Jansen MW, van Oers HA, Kok G, de Vries NK. Public health: disconnections between policy, practice and research. Heal Res Policy Syst. 2010 Dec 31;8(1):37.
Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med. 2007;34(2):105–14.
Leeman J, Sommers J, Vu M, Jernigan J, Payne G, Thompson D, et al. An evaluation framework for obesity prevention policy interventions. Prev Chronic Dis. 2012 Jun;9(6):E120.
Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. Interventions in Organizational and Community Context: A Framework for Building Evidence on Dissemination and Implementation in Health Services Research. Adm Policy Ment Heal. 2008 Mar;35(1–2):21–37.
Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011 Apr 23;6(1):42.
Moulton S, Sandfort JR. The Strategic Action Field Framework for Policy Implementation Research. Policy Stud J. 2017 Feb 1;45(1):144–69.
Rütten A, Lüschen G, von Lengerke T, Abel T, Kannas L, Diaz JAR, et al. Determinants of health policy impact: a theoretical framework for policy analysis. Sozial-und Präventivmedizin/Social Prev Med. 2003;48(5):293–300.
Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, et al. A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Adm Policy Ment Heal Ment Heal Serv Res. 2008 Mar;35(1–2):84–97.
Shortell SM. Increasing value: a research agenda for addressing the managerial and organizational challenges facing health care delivery in the United States. Med Care Res Rev. 2004;61(3 Suppl):12S-30S.
Spoth LA, Greenberg M, Leaf P, Brown CH, Fagan A, Catalano RF, et al. Addressing Core Challenges for the Next Generation of Type 2 Translation Research and Systems: The Translation Science to Population Impact (TSci Impact) Framework. Prev Sci. 2013;14(4):319–51.
Thomann E, Hupe P, Sager F. Serving many masters: Public accountability in private policy implementation. Governance. 2018 Apr 1;31(2):299–319.
Lukas CV, Holmes SK, Cohen AB, Restuccia J, Cramer IE, Shwartz M, et al. Transformational change in health care systems: An organizational model. Health Care Manage Rev. 2007 Oct;32(4):309–20.
Wandersman A, Alia K, Cook BS, Hsu LL, Ramaswamy R. Evidence-based interventions are necessary but not sufficient for achieving outcomes in each setting in a complex world: empowerment evaluation, getting to outcomes, and demonstrating accountability. Am J Eval. 2016 Dec 1;37(4):544–61.
Beidas RS, Stewart RE, Adams DR, Fernandez T, Lustbader S, Powell BJ, et al. A Multi-Level Examination of Stakeholder Perspectives of Implementation of Evidence-Based Practices in a Large Urban Publicly-Funded Mental Health System. Adm Policy Ment Heal. 2016;43(6):893–908.
Brodowski ML, Counts JM, Gillam RJ, Baker L, Collins VS, Winkle E, et al. Translating Evidence-Based Policy to Practice: A Multilevel Partnership Using the Interactive Systems Framework. Fam Soc J Contemp Soc Serv. 2013 Jul 3;94(3):141–9.
Brownson RC, Allen P, Jacob RR, Harris JK, Duggan K, Hipp PR, et al. Understanding Mis-implementation in public health practice. Am J Prev Med. 2015 May 1;48(5):543–51.
Cheadle R.; LoGerfo, J. P.; Schwartz, S.; Harris, J. R. A. E. Promoting Sustainable Community Change in Support of Older Adult Physical Activity: Evaluation Findings from the Southeast Seattle Senior Physical Activity Network (SESPAN). J Urban Heal. 2009;1–9.
Fleuren M, Paulussen T, Van Dommelen P, Van Buuren S. Towards a measurement instrument for determinants of innovations. Int J Qual Heal care J Int Soc Qual Heal. 2014;26(5):501–10.
Gotham HJ, White MK, Bergethon HS, Feeney T, Cho DW, Keehn B. An implementation story: Moving the GAIN from pilot project to statewide use. J Psychoactive Drugs. 2008;40(1):97–107.
Hargreaves M, Cole R, Coffee-Borden B, Paulsell D, Boller K. Evaluating Infrastructure Development in Complex Home Visiting Systems. Am J Eval. 2013 Jun;34(2):147–69.
Haug C, Rayner T, Jordan A, Hildingsson R, Stripple J, Monni S, et al. Navigating the dilemmas of climate policy in Europe: evidence from policy evaluation studies. Clim Change. 2010;101(3–4):427–45.
Horner RH, Kincaid D, Sugai G, Lewis T, Eber L, Barrett S, et al. Scaling Up School-Wide Positive Behavioral Interventions and Supports: Experiences of Seven States With Documented Success. J Posit Behav Interv. 2014 Oct 8;16(4):197–208.
Monroe-DeVita M, Morse G, Bond GR. Program fidelity and beyond: Multiple strategies and criteria for ensuring quality of assertive community treatment. Psychiatr Serv. 2012;
Painter K. Legislation of Evidence-Based Treatments in Public Mental Health: Analysis of Benefits and Costs. Soc Work Public Health. 2010;25(5):511–26.
Perla RJ, Bradbury E, Gunther-Murphy C. Large-Scale Improvement Initiatives in Healthcare: A Scan of the Literature. J Healthc Qual Promot Excell Healthc. 2013 Jan 1;35(1):30–40.
Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.
Powell BJ, Proctor EK, Glass JE. A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions. Res Soc Work Pr. 2014;24(2):192–212.
Powell B, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 Feb 12;10(1):21.
Rhoades BL, Bumbarger BK, Moore JE. The Role of a State-Level Prevention Support System in Promoting High-Quality Implementation and Sustainability of Evidence-Based Programs. Am J Community Psychol. 2012 Dec;50(3–4):386–401.
Rieckmann Anne E.; Cassidy, Elaine F.; McCarty, Dennis TR. K. Employing policy and purchasing levers to increase the use of evidence-based practices in community-based substance abuse treatment settings: Reports from single state authorities. Eval Program Plann. 2011;34(4):366–74.
Rieckmann T, Abraham A, Zwick J, Rasplica C, McCarty D. A Longitudinal Study of State Strategies and Policies to Accelerate Evidence-Based Practices in the Context of Systems Transformation. Health Serv Res. 2015 Aug 1;50(4):1125–45.
Rubin RMM, Hurford MOO, Hadley T, Matlin S, Weaver S, Evans AC. Synchronizing watches: The challenge of aligning implementation science and public systems. Adm Policy Ment Heal Ment Heal Serv Res. 2016 Nov 1;43(6):1023–8.
Yamey G. What are the barriers to scaling up health interventions in low and middle income countries? A qualitative study of academic leaders in implementation science. Glob Heal. 2012;8:11.
Zhang Y, Marsh D. Learning by Doing: The Case of Administrative Policy Transfer in China. Policy Stud. 2016 Jan 2;37(1):35–52.
VanDeusen Lukas C, Engle R, Holmes S, Parker V, Petzel R, Nealon Seibert M, et al. Strengthening organizations to implement evidence-based clinical practices. Health Care Manage Rev. 2010;35(3):235–45.
Pal LA. Beyond Policy Analysis: Public Issue Management in Turbulent Times. 5th ed. Nelson Education; 2014.

Table 1 – Search Terms

Implementation Terms		Government Level		Organizational Level		Practice Level		Evidence Terms (with and without dashes)
implement*	AND	policy	OR	organizational polic*	OR	“clinical guideline”	AND	“evidence-based practice*”
“knowledge translation”		strategy		policy and procedures manual		“practice guideline”		“evidence-informed practice*”
“knowledge mobili*”		“action plan”		procedures manual		scope of practice		“evidence- informed policy”
								“evidence-based policy”

Table 2a – Overview of Included Conceptual Literature

Author	Year	Author Location	Topic Area	Focus	Name of Model/Theory/Framework	Framework Type (3)
Aarons et al(12)	2011	USA	Public services for children & families	Implementation	Conceptual model of global factors affecting implementation in public service sectors	Determinants framework
Bauman et al(38)	2006	Australia (majority)	Physical activity	Supra-National	A Six-Step Framework for International Physical Activity Dissemination	Process model
Bowen et al(65)	2010	USA	HIV	Organizational/ program	Rogers-Rütten Framework	Determinants + evaluation framework
Bowen & Zwi(56)	2005	Australia	Public health	Knowledge translation	Evidence-informed Policy and Practice Pathway	Determinants framework
Bruns et al(66)	2008	USA	Children & youth	System (state)	No name per se but addresses dimensions of state EBP implementation effort	Determinants framework
Burris et al(57)	2012	USA	Public health	System (law)	No name per se but unified framework integrating public health law and public health systems and services	Determinants framework
Campos & Reich(67)	2019	USA	Health	Policy, politics & stakeholders	No name but identifies six different directions for different stakeholders	Process model
Chaudoir, Dugan & Barr(9)^*	2013	USA	Health	System (measures of determinants)	A multi-level framework predicting implementation outcomes	Determinants framework
Cherney & Head(68)	2011	Australia	Evidence-based policy/practice	System	Components of a Support Delivery System: ‘9Cs’	Determinants framework
Chin & Goldmann (69)	2011	USA	Health	System	A Conceptual Model for Specifically Addressing Disparities 6 Key Levels of Influence	Implementation theory
Damschroder et al(7)	2009	USA	Health	Organizational	Consolidated Framework for Advancing Implementation Research (CFIR)	Determinants framework
Domitrovich et al(70)	2008	USA	Schools	Implementation quality	No name per se but identified as factors that can affect implementation quality: a multi-level model	Determinants framework
Evans & Davies(37); Dolowitz & March(36)	1999 2000	UK UK	Policy transfer	Policy	Policy transfer	Determinants framework
Feldstein & Glasgow(71)	2008	USA	Healthcare	Research to practice implementation	PRISM (Practical, Robust Implementation and Sustainability Model)	Process model
Fleuren, Wiefferink & Paulussen (72)	2014	Netherlands	Healthcare	Organizational/ program	No name per se but “Framework representing the innovation process and related categories of determinants”	Determinants framework
Godfrey(73)	2011	USA	Mental health	System	Hypothesized factors that influence ACT implementation	Determinants framework
Green et al(74)	2006	USA	Physical activity	Knowledge translation	Push-Pull Capacity Model	Process model
Greenhalgh et al(26)	2004	UK	Healthcare	Organizational	Diffusion of Innovations in Service Organizations	Determinants framework
Greig, Entwistle & Beech(75)	2012	UK	Healthcare	Implementation activity/practices	Activity Theory	Classic theory
Harris et al(76)	2012	USA	Health promotion	Organizational	Health Promotion Resource Center Dissemination Framework	Process model
Harvey & Kitson(77)	2016	Australia	Health services	Implementation	Integrated Promoting Action on Research Implementation in Health Services (I-PARIHS)	Determinants framework
Hendriks et al(78)	2013	Netherlands	Public health (childhood obesity)	Policy	Behavior Change Ball	Implementation theory
Hill & Hupe(52)	2003	UK & Netherlands	Policy implementation	Policy	No model/theory or framework but discussed ‘the multi-layer problem’	N/A
Hill & Hupe(45)	2002	UK & Netherlands	Policy implementation	Policy	N/A (book)	Determinants framework
Hodges & Ferreira(79)	2013	USA	Children & families	Policy (local)	Multilevel framework for local policy development and implementation	Determinants framework
Howlett(42)	2004	Canada	Policy implementation	Policy (instruments)	N/A	Other (most closely resembles Classic Theory)
Hupe(80)	2011	Netherlands	Policy implementation	Explaining policy implementation	Thesis of incongruent implementation	Determinants framework
Hupe & Hill(81)	2016	Netherlands & UK	Policy implementation	Policy	N/A	N/A
Jansen(82)	2010	Netherlands	Public health	Disconnections between policy, practice and research	3 niches of public health	Process model + determinants framework
Jilcott et al(83)	2007	USA	Public health	Evaluating policy implementation	Applying the RE-AIM framework to assess the public health impact of policy change	Evaluation framework
Johansson (11)	2010	Sweden	Human services	Policy	N/A	N/A
Leeman et al(84)	2012	USA	Obesity prevention	Policy	Center TRT’s evaluation framework	Evaluation framework
Lipsky(13)	1980	USA	Social services	Policy & individual	Street-Level Bureaucracy	Implementation theory

Matland(44)	1995	USA	Policy implementation	Policy	Ambiguity-Conflict Model of Implementation	Implementation theory
Mendel et al(85)	2008	USA	Mental health	Organizational/ community	Framework of Dissemination in Health Services Intervention Research	Process framework (2^nd)
Michie(86)	2011	UK	Behaviour change (EBPs)	Individual	The Behaviour Change Wheel	Implementation theory + determinants framework
Moulton & Sandfort(87)	2017	USA	Public service interventions	Policy	The Strategic Action Field Framework	Implementation theory
Pettigrew & Whip(55)	1992	UK	Business	Organizational/ firm	Understanding strategic change: three essential dimensions (Warwick Framework)	Classic theory
Proctor et al(47)	2011	USA	Mental health	Implementation outcomes	Conceptual Model of Implementation Research	Evaluation framework
Raghavan, Bright & Shadoin(59)	2008	USA	Mental health	Policy	A Policy Ecology of Implementation	Determinants framework
Rutten et al(88)	2003	Germany/ Europe	Health promotion	Policy	Determinants of policy analysis	Determinants framework + classic theory
Sabatier & Mazmanian (14)	1980	USA	Policy implementation	Policy	Framework of Analysis for the Implementation of Public Policy	Determinants framework + process model
Schoenwald et al^*(89)	2008	USA	Mental health	System	Conceptual model for the MacArthur research network on youth mental health child STEPs initiative on evidence-based practice in clinics and systems	Determinants framework
Shortell(90)	2004	USA	Health care	System	N/A Levels and associated assumptions about change	Implementation theory
Spoth et al(91)	2013	USA	Public health/ prevention	Population	Translation Science to Population Impact (TSci Impact) framework	Process model
Strehlenert (60)	2015	Sweden	Health and social care	Policy	Conceptual Model for Evidence-Informed Policy Formulation and Implementation	Process model
Thomann et al(92)	2017	Germany	Policy implementation	Policy	Extended Accountability Regimes Framework	Implementation theory
VanDeusen Lukas et al(93)	2007	USA	Heath care	Organizational	Framework for Organizational Transformation	Classic theory
Viennet & Pont(50)	2017	International	Education	Policy	Education Policy Implementation Framework	Determinants framework & implementation theory
Wandersman et al(94)	2016	USA	Empowerment evaluation	Innovation & system interface	Getting to Outcomes	Process model
Wisdom et al(61)	2014	USA	Innovation adoption	System	N/A	Determinants framework

^* also included in empirical literature

Table 2b – Overview of Included Empirical Literature

Author	Year	Author Location	Topic Area	Level of Focus	Methodology
Bax, de Jong & Koppenjan(46)	2010	Netherlands	Road safety	System	Policy analysis
Beidas et al(95)	2016	USA	Mental health	Stakeholder	Qualitative interviews
Brodowski et al(96)	2013	USA	Child abuse prevention	System	Descriptive case study
Brownson et al(97)	2015	USA	Public health	System (state + local)	Cross-sectional survey
Chaudoir, Dugan & Barr^*(9)	2013	USA	Health	System (measures of determinants)	Systematic review & criterion-validity assessment
Cheadle et al(98)	2009	USA	Physical activity promotion	Community	Evaluation – uncontrolled prospective design
Culotta, Wiek, & Forrest(53)	2016	USA	Climate change	Regional	Case study/policy analysis
Evans(54)	2013	UK	Health	Policy	Mixed methods survey & in-depth interviews
Fleuren et al(99)	2014	Netherlands	Prevention child health care/schools	Innovation determinants	Systematic review + Delphi study
Gotham et al(100)	2008	USA	Addictions	System (state)	Case study
Grace et al(48)	2015	Australia	Mental health	Policy	Policy analysis (document analysis)
Grundy & Smith(49)	2011	Canada	Employment	Policy	Policy analysis
Hargreaves et al(101)	2013	USA	Home visiting	Systems	Mixed methods
Haug et al(102)	2010	Europe	Climate change	Policy	Literature review
Horner et al(103)	2014	USA	School behavioral supports	Multi-state EIPP	Descriptive evaluation
Monroe-DeVita et al(104)	2012	USA	Mental health	EIPP	Literature Review
Painter(105)	2010	USA	Mental health	Policy	Single case study (document analysis + secondary data analysis of single provider)
Perla, Bradbury & Gunther-Murphy(106)	2013	US & UK	Healthcare	System	Scan of literature using modified Delphi technique
Powell et al(107)	2012	USA	Health & mental health	EIPP	Narrative review
Powell et al(108)	2014	USA	Mental health	EIPP	Systematic review
Powell et al(109)	2015	USA	Health & mental health	EIPP	Delphi process
Rhoades et al(110)	2012	USA	Prevention (of crime & delinquency)	System (state level)	Case description
Rieckmann(111)	2011	USA	Addictions	Policy	Mixed methods (survey & key informant interviews)
Rieckmann(112)	2015	USA	Addictions	Policy	Mixed methods (survey & key informant interviews)
Rubin(113)	2016	USA	Alignment of implementation & public systems	Systems	Intro to special issue (review of articles)
Schoenwald et al^*(89)	2008	USA	Mental health	System	Structured survey (national sample)
Yamey(114)	2012	USA	Health in LMICs	System	Key informant interviews
Zhang & Marsh(115)	2016	China	Administrative policy transfer	Policy	Policy analysis
National Implementing Evidence-Based Practices Project articles (53 sites; 8 states), n = 9**
Bond et al(27)	2009	USA	Mental health	System (multi-state)	Mixed methods
Finnerty et al(28)	2009	USA	Mental health	Policy/System	Instrument development & testing
Isett et al(29)	2007	USA	Mental health	System (multi-state)	Qualitative (interviews)
Isett et al(30)	2008	USA	Mental health	System (multi-state)	Case study (site visits + semi-structured interviews)
Jones et al(31)	2014	USA	Mental health	System (multi-state)	Semi-structured interviews (state leaders)
Mancini et al(32)	2009	USA	Mental health	Innovation (2 states)	Mixed methods (fidelity measurement + interviews, surveys & site visits)
Peterson et al(33)	2014	USA	Mental health	System (multi-state)	Longitudinal analysis (descriptive)
Rapp et al(34)	2005	USA	Mental health	System (multi-state)	Descriptive
Rapp, Goscha & Carlson(35)	2010	USA	Mental health	System (state)	Descriptive

^* also included in conceptual **9 articles described individually in subsequent rows

Table 3 – Policy-related strategies and examples of those strategies for implementation according to type of target

Target

Strategy

Examples

References

System

Policy authority (governance arrangement)

Centralization/decentralization of policy authority (e.g., creating a regional infrastructure with some policy authority to oversee implementation)
Accountability of the state sector’s role in implementation (e.g., develop system-wide performance indicators or targets, monitor performance and fidelity, evaluate, report results publicly, consider enforcement strategies)
Leadership for implementation (through the appointment of state sector leaders, dedicated resources, garnering support for innovation and its implementation)
Stewardship of the non-state sector’s role in implementation (e.g., constructing formal opportunities for non-state sector in oversight of implementation; contracting with non-state sector for implementation-related activities; fostering networks and linkages across different types of organizations who are engaged in implementation)

(7,12,66,109,27–31,37,56,61)

Funding system infrastructure

(financial arrangement)

Dedicate resources for system infrastructure to support implementation (e.g., intermediaries, technical assistance centres, backbone organizations, facilitators, etc.)
Create funding sources that align with time needed for effective implementation and scaling

(9,27,96,103,106,109,110,28,31,32,35,66,73,91,95)

Re-designing system to meet needs

(delivery arrangement and implementation-related supports)

Consider impacts of implementation on availability of care/service and plan for scaling-up across the geographical area or population
Assess possible impacts on other services (e.g., wait times, etc.) in response to implementing innovation

(7,14,105,109,26,34,65,74,77,78,90,100)

With what supports service is provided (delivery arrangement)

Create or change system-wide record systems or information and communication technologies to support implementation

(29,103,106,109)

Organization

Organizational authority

(governance arrangement)

· Management approaches in support of optimal implementation, including: developing data collection systems, developing and monitoring performance indicators, quality improvement plans, use of scorecards, or public reporting
· Develop and deploy appropriate organizational leadership for implementation oversight and engagement
· Include innovation as part of accreditation processes
· Engage in networks/multi-institutional arrangements in support of implementation

(7,34,89,100,106,109)

Funding organizations

(financial arrangement)

Provide service grants or contract with organizations to support implementation or to offset additional administrative costs of implementing an innovation (e.g., training, data infrastructure changes, workforce stability impacts, etc.)
Prospective payments to cover lag-time costs when beginning to implement an innovation
Targeted payments or penalties based on organizational performance related to innovation (e.g., changing reimbursement rate structure so that providers of high fidelity receive modestly higher per unit rate)
Targeted payments or penalties based on client outcomes
Shift organizational funding models to support implementation (e.g., from fee-for-service to no-risk managed care arrangements)

(12,28,29,34,35,59,89,103,105,109,112,113,116)

Where service is provided

(delivery arrangement)

· Adjust sites of service delivery in response to an innovation
· Consider how the physical structure, facilities & equipment can support innovation during implementation and ensure appropriate supply (supply chain management)
· Adjustments to the organizational scale in response to an innovation (e.g., number of beds, units of service, etc.)

(48,109)

With what supports service is provided (delivery arrangement)

· Change organizational record systems or other information and communication technologies to support implementation

(106,109)

Organization-targeted implementation supports

(Implementation-related supports)

· Develop educational materials, hosting educational meetings, training or outreach visits tailored to organizations
· Develop and disseminate program or organizational service standards
· Provision of technical assistance and other forms of implementation support
· Support development and maintenance of inter-organizational collaboratives, communities of practice and other forms of inter-organizational communication/learning
· Consider non-monetary awards, incentives and disincentives for organizations (e.g., exemplary program award)

(7,9,109,28,29,35,53,76,77,85,95)

Workforce/

Service Provider

Professional authority

(governance arrangement)

· Create or alter training & licensure requirements
· Change scope of practice to reflect innovation
· Alter where providers can practice geographically and in what systems (public vs private, etc.)
· Continuing competence (e.g., provide training & continuing education unit credits for innovation or disallow certain courses for credit)
· Professional liability (e.g., change liability laws)
· Alter university curricula to include knowledge of innovation

(27,28,34,59,89,100,104,109)

Remunerating providers

(financial arrangement)

· Reimbursement for program participation, extra efforts in applying the innovation or lost time due to training
· Increase reimbursement rate
· Changing the way providers are reimbursed to encourage implementation (e.g., from fee-for-service to capitation)
· Loan forgiveness
· Targeted payments or penalties for performance
· Targeted payments or penalties based on outcomes
· Review and align fiscal and billing policies and incentives for providers
· Make billing easier for providers

(7,27,76,85,90,95,109,113,32,34,47,48,51,59,61,71)

By whom service is provided

(delivery arrangement)

· Assess and improve workplace conditions for providers to foster implementation
· Extend the role of a particular provider within their existing scope of practice
· Shift tasks between types of providers
· Optimize the performance of the workforce in their current roles by creating, disseminating and monitoring guidelines or standards of care for service providers

(7,37,73,78,79,86,94,100,109)

Workforce-targeted implementation supports

(implementation-related supports)

· Develop educational materials, hosting educational meetings, training or outreach visits
· Engage local opinion leaders
· Reminders & prompts
· Audit & feedback
· Coaching
· Develop either tailored or multi-faceted interventions to support implementation
· Consider non-monetary awards, incentives and disincentives for workforce

(28–30,34,59,66,85,96,100,109)

Consumer

Consumer & stakeholder involvement

(governance arrangement)

Consumer protection (laws, complaints management, etc.)
· Consumer, family & stakeholder engagement in implementation & monitoring

(59,89,100,109)

Incentivizing consumers

(financial arrangement)

· Alter consumer/patient fees
· Consider disincentives that may exist for consumers to be successful (e.g., some employment programs)
· Subsidies for private health insurance

(29,48,109)

Consumer-targeted implementation supports

(implementation-related supports)

· Information or education provision
· Behaviour change support
· Skills and competencies development
· Communication and decision-making facilitation

(37,59,78,86)

Innovation

Commercial authority

(governance arrangement)

· Adjust licensure & registration requirements to support implementation
· Consider pricing & purchasing
· Establish voluntary agreements on advertising

(78,86)

Purchasing products & services

(financial arrangement)

Changes to the scope and nature of insurance plans: extending or ending insurance coverage
Adjust list of covered/reimbursed services & products
Change restrictions or caps on coverage/reimbursement for innovation and related supports
Change mechanisms for billing
Prior approval requirements

(32,59,73,89,94,97,100,104,109)

Table 4 – Determinants of implementation from a policy perspective and the factors that characterize the determinants

Determinant	Description	Factors that characterize determinant
Characteristics of the EIPP	Not possible to predict the success or failure of a particular policy package based on its intrinsic characteristics alone(45) Need to examine questions such as whether the policy selected: is an appropriate fit with the problem(49,50) aligned with existing context(12,51)	Relative advantage(26,65) Compatibility(26,65) Complexity of goals and ease of implementation(26,65,87) Obligations(26,65) Resources(26,65) Existing relationship with state and provider(112) Level of ambiguity of the EIPP(44) Level of conflict among stakeholders(44) Interaction of policy characteristics with other determinants(45)
II. Policy Formulation Process	Shape given to a policy by the initial formation processes has an impact on its implementation(45) Depending on the implementation approach, the government may distribute responsibility for some or almost all of the policy formulation process to other stakeholders(12) Level of involvement of service organizations, street-level bureaucrats and recipients may influence the confidence in, and support of, the policy decision and improve the chances for successful implementation(106)	Government actors responsible for formulating policy(45) Perceived legitimacy of government actors(45) Extent to which there is opportunity to provide feedback(45) Responsiveness of policymakers to feedback(45) Level of involvement of non-governmental actors(12,50,106) Adequacy of planning for implementation (consideration of resources for implementation)(106) Constraints experienced during formulation(12,26)
III. Vertical Public Administration and Thickness of Hierarchy	Vertical Public Administration: Term used to identify the layers in the policy transfer process Refers to separate governments exercising their authority with relative autonomy(45) Policies generated outside of a socio-political level may be more or less acceptable to that level Thickness of Hierarchy: Number and complexity of institutions, departments or agencies at a particular socio-political level The thicker the hierarchy, the more managerial competence, professionalism and governance skills are required by public servants in order to support effective implementation(80)	Number of socio-political levels(45) Acceptability of policy generated outside of a particular socio-political level(115) Appropriateness of socio-political level(53) Thickness of each socio-political level (number and complexity of institutions, departments or agencies and their coordination and interdependence)(52)
IV. Networks/Inter-Organizational Relationships	Reflects the existence and nature of the relationships between parallel organizations who must collaborative in order to achieve effective implementation and who do not have a hierarchical relationship(45) Better connections among stakeholders also increases the opportunity for rapid diffusion and informal spread of innovation, facilitating implementation	Degree of coordination among: Systems(61) Organizations(53) Donors /other funders(97) Leaders(12) Formality (formal or informal)(89) Network type (e.g. policy or inter-organizational)(77) Coherence and strength of connections(117)
Implementing Agency Responses	Factors affecting the responses of implementing agencies can be divided into: issues related to the overall characteristics of the agencies, and behaviour of front-line or street-level staff(45) Overall “health” of organizations and how front-line/street-level bureaucrats use their discretion and power impact implementation success	Overall characteristics of the agencies: Level of organizational control(45,92) Rate of staff turnover(51) Organizational decision-making processes(51) Extent of policy and behaviour-related change required(14,87) Attitudes of the agencies(14,71) Resources of the agencies (e.g., minimum “investment threshold” in implementation infrastructure(106) or cost-absorptive capacity of agency to absorb additional costs associated with implementation(12) or certainty of funding(73)) Impetus for change((93) (e.g., external mandates may increase an agency’s predisposition (i.e. motivation), but not its capacity to adopt an innovation; mandates may divert activity away from other innovations or locally generated priorities(26)) Perception of implementation approach (e.g. if approach is punitive, mandatory or “top down”)(95) Behaviour of front-line or street-level staff Level of discretion and level of relative autonomy from organizational authority affect the amount of interpretation of EIPP(13,92) Competing accountabilities (e.g., state, market, professional, societal)(92) Power distribution between actors at the front-line, agency and political levels(30) Personal characteristics including their knowledge, skills, and perceived support from colleagues(51)
VI. Attributes and Responses from Those Affected by EIPP	The characteristics of the people affected by the EIPP, their response to it, and the impact of the responses Most evident when those affected are powerful, such as in regulatory policy when those regulated are large organizations(45)	Diversity of target group behaviour(14) Target group as percentage of the population(14) Impacts on stability of the workforce and responses to instability(12)
VII. Timing/ Sequencing	· Implementation processes at scale require adequate time, which doesn’t always align with the cycles they are subject to and some authors have identified the lack of time or short-term opportunism as barriers to effective implementation(53,54) · The sequencing of activities and alignment of implementation with other cycles is also important	Balance of predictability and adaptiveness to changing circumstances(55,102) The simultaneous address of system levers (including policy changes, measurement systems, and regulatory mechanisms)(106) Timing and pace of cycles, such as political, policy and funding cycles(113) Specific aspects of time that impact implementation: the phased structure of the implementation process(113) when and how the implementation efforts are initiated(113) timeframes for funding and leadership support(113) the need to demonstrate the impacts early return on investment of time and money(113)
VIII. External Environment or Policy Context	Factors outside of the policy area of focus may influence implementation Can be referred to generally as the “political and social climate”, as unmodifiable or macro “context” or as “socio-economic conditions”(9,14,56–61) While most included articles did not address these determinants in depth, an overall examination of extracted data suggested two theoretical frameworks would be useful for classifying and understanding these determinants: A) 3I+E framework that identifies the Institutions, Interests, Ideas and External events that help explain what influences policy choices(62) B) Taxonomy of health and social system arrangements classified according to the governance, financial and delivery arrangements(63) These broader context and system arrangements may be critically important in explaining implementation outcomes and these frameworks provide some logic and organization to potential variables	3I+E framework Ideas (e.g., the interplay between beliefs and values of policymakers and research evidence in a general way(56)) Interests (e.g., the political culture and the depth of social cleavages(42)) Institutions (e.g., relevant policies from other areas that “may represent potentially powerful contextual effects”(50,91) External factors (e.g., technology and technological changes,(14,57)) economic forces operating in the overall society,(81) and environmental (in)stability(77,87)) Taxonomy of health and social system arrangements.(63) Governance arrangements that are not specific to the EIPP being implemented but are still relevant to understanding implementation outcomes (e.g., centralization and power distribution of government(30,89) or the form of governance structures (omnibus/discrete)(89)) Financial arrangements (e.g., private/public contractual relations, reimbursement rates and mechanisms,(89) and existing resource distribution(30)) Delivery arrangements - referred to more generally in the health-focused articles as “health(care) system and services context”(56,60) or “medical delivery system”(57)

Table 5 - Types of policy actors identified in implementation

Actor	Sub-Type (non-exhaustive)	Role Description	Role in Implementation (non-exhaustive)	References
Political Actors	Politicians	· Represent citizens (in a democracy) through popular consensus. · Mandate to create laws and policies with varying levels of authority · Can be supra-national, national/federal, state/provincial, regional, local/municipal	· Most important level of elected officials is the level where most policy authority rests for area of implementation Develop & pass laws/policies (e.g., mandating a particular EIPP) Provide leadership and focus Source of funding for implementation (organizations, providers, and/or consumer levels)	(34,38,96,103,104,110,61,67,74,78,82,84,85,95)
Political Actors	Other Elected Officials	Similar to elected politicians but mandate is limited to a particular policy domain and (often) limited geographic jurisdiction (e.g., sheriff, judge, school board trustee, etc.)	If policy authority rests at their level, they may develop & pass laws Enforce laws/polices from other levels Interpret/adapt laws/policies for their implementation Provide leadership & focus Source of funding for implementation (organizations, providers, and/or consumer levels)	(78)
Bureaucratic Actors	Executive Departments	Departments or ministries who specialize in a unique area of government responsibility (e.g., health) Responsible for carrying out the “vision” of an elected official with leadership for that portfolio (e.g., Minister of Health) Not elected nor formally tied to a particular political party	Support policy development, including implementation considerations Operationalize policy/law passed by politicians May allocate tasks, responsibilities and define competencies for implementation Monitor policy implementation and track outputs or outcomes Source of funding for implementation (organizations, providers, and/or consumer levels)	(34,82,95,97,100)
	Boards and Agencies of Government	Often operate semi-independently from government but are appointed by them In most cases, they deal exclusively with one particular sub-field of responsibility in which the demand for public services is especially high (e.g., food inspection agency, state mental health authority)	Regulation & enforcement Interpretation of policies/laws Monitor policy implementation & track outputs or outcomes May have the ability to apply penalties for non-compliance May allocate tasks, responsibilities & define competencies for implementation	(27,28,34,61,85,95)
	Self-Governing Regulatory Agencies	Bodies that regulate the conduct of their own members (such as admissions and discipline) and are empowered to do so by the appropriate level of government and their members (e.g., medicine, law, etc.) Regulators are drawn from the membership	Can set or change: scope of practice, training & licensure requirements, or professional liability to support implementation Develop/adopt guidelines or standards Monitor quality and safety and continued competence of professionals during implementation	(95)
	Judicial System	System of courts that provide a formal mechanism for interpretation and application of laws in the name of the state and resolves disputes	Interpret/re-interpret laws through rulings that may affect how they are implemented Define/re-define public policies through legal challenges	(57)
Special Interests	Implementing Agencies	Organizations or programs that are responsible for implementing the laws or policies developed (e.g., hospitals, schools, child welfare agencies, industry, etc.) Location(s) where the majority of the implementation takes place	Interpretation of policies/laws Develop or adapt organizational policies & procedures to support implementation Training & support for workforce Provide or manage funds to support implementation Monitor & evaluate implementation at organizational level	(28,34,110,49,54,67,69,74,96,100,104)
	Street-level Bureaucrats	The schools, police and welfare departments, lower courts, legal services offices, and other agencies whose workers interact with and have wide discretion over the dispensation of benefits or the allocation of public sanctions(13) Have 1) relatively high degree of discretion; 2) relative autonomy from organizational authority(13)	Interpretation of policies/laws Often the parties responsible for changing their behaviours or practices during implementation	(13,67,69)
	Insurers	Organizations or government bodies that manage risk by pooling risk across a group of individuals and providing coverage to them for needed services Managed care organizations are a specific type of insurer in health care that monitor and control the provision of care in an effort to increase quality through regulating the choices of providers and patients	Have the ability to change the risk pool by insuring more or fewer people (scope and nature of insurance plan) Can adjust the list of covered/reimbursed organizations, providers, services & products Can change billing/reimbursement processes to facilitate implementation Engagement & potential influence with political & bureaucratic actors (feedback loops) regarding implementation & scaling	(74,85,95)
	Donors/ Foundations	Organizations that raise and allocate funds based on a specific mandate that they identify	Funding and/or in-kind implementation supports (e.g., human resources) May have funded an innovation and now have a vested interest in seeing it implemented or scaled (bring leadership & focus, implementation & scaling expertise, etc.) Engagement & potential influence with political & bureaucratic actors (feedback loops) to support implementation & scaling	(67,114)
	Government Corporations	Organizations or businesses that are run independently from government but are still ultimately accountable to them	Interpretation of policies/laws Develop or adapt organizational policies and procedures to support implementation
	Unions	Organized associations of workers created to promote and protect their interests in the workplace	Negotiate contractual relationships with implementing organizations on behalf of members (can influence the ease of implementation) Engagement & potential influence with political & bureaucratic actors (feedback loops) regarding implementation & scaling	(34,67)
Experts	Scientists/ Researchers	Individuals or research programs that systematically gather, analyze and use research and other evidence through processes such as theorizing, synthesizing, and hypothesis testing, to gain and share understanding and knowledge	Share or contribute research expertise concerning the problem, the innovation, the implementation or the evaluation of the implementation effort & any expected outcomes Engagement & potential influence with political & bureaucratic actors (feedback loops) to support implementation & scaling	(35,110)
	Field or Practice Leaders/ Champions	· Individuals who belong to a service providing community and are viewed as leaders or champions of an innovation and its implementation	· Share or contribute practice expertise concerning the problem, the innovation, the implementation or the evaluation of the implementation effort & any expected outcomes · Act as champions for implementation to members of their service providing community and to other policy actors Engagement & potential influence with political & bureaucratic actors (feedback loops) to support implementation & scaling	(31,56,68)
	Patients or Persons with Lived Experience & Families/Carers	· Individuals who bring personal knowledge or experience of a problem, condition, or service and who are the intended beneficiaries or ultimate “targets” of implementation, and/or · Individuals who are family members or carers to individuals who bring personal knowledge or experience of a problem, condition or service	· Share or contribute lived experience of the problem, the innovation, the implementation or the evaluation of the implementation effort & any expected outcomes	(28,35,37,69,79,96,109)
	Innovation/ Developers and Disseminators	· Organizations, programs or individuals who have developed a process, program or product to be implemented	· Synthesize knowledge about innovation & package it in ways that are “usable” · Actively seek opportunities for innovation to be adopted in policy and/or practice · Provide expertise about the innovation during implementation process · Adapt innovation and materials as needed during implementation process	(76,110)
	Intermediaries and technical assistance providers	Organizations, programs or individuals that work “in between” policymakers, funders, and frontline implementers, to facilitate effective implementation drawing on expertise in implementation Also known as purveyor organizations, backbone organizations or central bodies charged with coordination	Translate policy intention for implementing agencies Provide technical assistance to implementing agencies (e.g., guidance on implementation process, coaching, decision support, monitoring & evaluation) Provide mechanism for communication between service delivery, policy systems and innovation developer (if applicable)	(9,31,110,37,53,68,70,77,96,100,109)
Other	Media	· Individuals and organizations that communicate information through a variety of channels, including formal media outlets and social media outlets	Monitor implementation and communicate facts or perceptions of the process and outcomes to the public Provides feedback loop for political actors, bureaucratic actors, special interests and experts regarding implementation	(34,85)

bullocketalCISImpSciprotocol.docx

Download PDF

Journal Publication

published 15 Feb, 2021

Read the published version in Implementation Science →

Editorial decision: Accept
05 Jan, 2021
Editor assigned by journal
03 Jan, 2021
Submission checks completed at journal
03 Jan, 2021
Editor invited by journal
03 Jan, 2021

You are reading this latest preprint version

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

Status:

Journal Publication

Version 3

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Discussion

Conclusions

Declarations

Ethics approval and consent to participate:

Consent for publication:

Availability of data and materials:

Competing interests:

Funding:

Authors’ contributions:

Acknowledgements:

References

Tables

Supplementary Files

Status:

Journal Publication

Version 3