Search results and article selection
Our database search retrieved 16,107 documents and 11,434 unique documents once duplicates were removed. The review of titles and abstracts was completed independently by two reviewers on a random sample (n = 171) of the documents. The Kappa score was 0.72 indicating substantial agreement. Figure 1 provides a flow diagram outlining the search strategy. Following these criteria for the remaining titles and abstracts resulted in 1,208 documents included for full text review. The full text review excluded an additional 940 documents leaving 268 potentially relevant documents (excluded documents and the rationale for exclusion are available upon request). Of these, 23 conceptual documents, 243 empirical documents and two documents that included both conceptual and empirical elements were included for the data extraction and analysis phase. We sampled and extracted data on all of the conceptual articles. For the empirical articles, we chose a maximum variation sampling approach based on the subject matter and article topic with an initial sample of 10% of the articles. We also noted that nine of the articles related to a large, multi-year national implementation study (27–35). Because this was the largest and most comprehensive account of the role of policy in large-scale implementation efforts identified through our search, we included these as a sub-group for data extraction. This approach led to data extraction for 34 empirical articles.
In addition to these two approaches we sampled articles that filled in conceptual gaps as our model developed. This process resulted in the retrieval of an additional 26 conceptual articles and 3 empirical articles. In total, 86 unique documents were included with two of these documents used in both the conceptual and empirical data extraction (Tables 2a and 2b). While our process was inclusive of English language publications from any country, the majority of articles were conducted by US researchers (n = 57), with the others coming mainly from other Western countries (the United Kingdom (n = 8), Netherlands (n = 7), Australia (n = 5), Canada (n = 2), Sweden (n = 2), Germany (n = 2), and Europe, China and OECD (n = 1)). Articles covered a range of topics including health and health care, public health, mental health and addictions, children and youth, social care, justice, and climate change, among others. The conceptual documents included all of the categories of theories, models and frameworks identified by Nilsen (3), with the Determinants Framework type being most common. The empirical articles employed a wide array of methods that fall into the broad categories of qualitative, quantitative and mixed methods.
Through this process we noted several general observations regarding the characteristics of existing literature. In terms of the scholarly disciplines, most of the implementation science literature focused on the organizational or service provider levels with an emphasis on changing practice, often by introducing an Evidence-Informed Policy or Practice (EIPP). The knowledge translation literature included policymakers as a target audience for research evidence, but the focus was on the agenda setting or policy formulation stages of the policy cycle, as opposed to the implementation of an EIPP. Here, the scholarship focused on strategies to increase the use of evidence in policy decision-making. The public policy literature included theory describing “top-down”, “bottom-up” and integrated approaches to implementing an EIPP. The object of implementation in this area was the policy itself, rather than a specific program or practice. There was often no clear articulation of independent and dependent policy-related implementation variables across any field, although many articles did partially address this.
How policy is described in implementation theory and processes
Our coding based on the compass question resulted in the following characterization of how policy is described in implementation theory and processes:
Policy is described as:
- Context in which implementation occurs (i.e., only briefly citing a policy as the reason for implementation)
- Focusing lens, signaling to systems what the priorities should be (i.e., referring to policy statements or attention by policymakers as a signal about what is important to prioritize)
- Innovation itself – the implementation object (i.e., the “thing” being implemented is policy, such as new legislative policy on tobacco cessation).
- Lever of influence in the implementation process (i.e., policy is identified as at least one of the factors influencing the implementation process)
- Enabler/facilitator or barrier to implementation (mediating variable) (i.e., while policy is identified as being external to the implementation effort, it is later found to be a barrier or facilitator to implementation)
- Outcome – the success of the implementation process is at least partially defined and measured by a change in policy
- Policy actors as important participants or leaders in implementation
Our approach to developing the theoretical framework was two-fold. The findings from our analysis suggested constructs that addressed both the process of implementation and the factors underpinning the success or failure of implementation. We therefore first developed a process model (3) that describes the steps in the process of translating EIPPs into effectively embedded system changes. Next, we developed a determinants framework, which specifies the types of policy determinants (independent variables) that affect implementation outcomes (dependent variables). This two-part theoretical framework achieves two goals: 1) the process model is most useful in describing the process of implementation from a policy perspective, and 2) the determinants framework is most useful for understanding and explaining policy-related influences on implementation outcomes.
Part 1 – Process Model
Figure 2 depicts this novel process model focusing on one policy or system level. What follows is a narrative description of the model.
Policy is shaped as it moves through systems. The process through which policy travels from one level to another is known as policy transfer (36–38). Each policy level is nested in a context that includes existing ideas (values, evidence, etc.), interests (interest groups, civil society, etc.), institutions (existing rules and institutional structures) and external factors (natural disaster, change in economic conditions) that affect the interpretation of the policy package (39,40). This context affects how a problem is defined, whether it has the attention of decision makers and whether it is up for active decision-making. This aligns with the “problem definition” and “agenda setting” stages of the policy cycle but is also described as part of the “exploration phase” in implementation science (12,41). Once a decision has been reached that something should be done to address a given issue, attention shifts to the “policy development” stage of the policy cycle, which aligns with the “adoption decision and preparation” stage of implementation. It is during the policy development/adoption decision and preparation stage that the policy package gets developed.
A policy package usually includes a mix of policy levers or instruments, including legal and regulatory instruments, economic instruments, voluntary instruments, or information and education instruments (42,43). The policy package can also include some implementation guidance such as a description of the overall implementation strategy architecture, the major streams of activity, timing of events and milestones, and, roles and responsibilities.
The level of ambiguity of the policy package in terms of its goals and means of attaining them, and the amount of conflict among actors with respect to the policy package are important to help characterize the implementation process and to explain its outcomes. According to Matland (44) the consideration of ambiguity and conflict leads to four types of implementation processes: 1) Administrative implementation occurs when there is low policy ambiguity and low policy conflict (e.g., eradication of small pox); 2) Political implementation occurs when there is low ambiguity but high levels of conflict (e.g., public transit); 3) Experimental implementation occurs when there is high ambiguity but low conflict (e.g., Head Start programs for young children); and 4) Symbolic implementation occurs when both ambiguity and conflict are high and policies only have a referential goal and differing perspectives on how to translate the abstract goal into instrumental actions (e.g., establishing youth employment agencies).
The policy implementation process can start at any level, move in any direction and can “skip” levels. Power also shifts as implementation proceeds through levels (29,45). The level with the most implementation activity tends to have the most power. This is true not only for different levels of governance, but as implementation cascades across organizations, through “street level bureaucrats”(13) and on to the end-user or target population (the “recipient”) of the implementation process. Policy decisions at one level becomes context for other levels. Implementation activities at one level can exert either direct or indirect effects on another level. The context surrounding each level (prevailing ideas, interests, institutions and external events) influences the acceptability and ultimate success of implementation. Finally, the overall implementation approach may need to shift over time in response to a constantly evolving context. For example, one study found it necessary to change the implementation approach for a road safety program in respond to changes in policy authority (46).
The process of implementation is undertaken in order to lead to outcomes, which can be separated and measured at different levels. Proctor and colleagues (47) identifies three separate outcomes: 1) implementation outcomes; 2) service outcomes; and 3) recipient-related outcomes. Along with these outcomes, our model includes policy- and systems-level outcomes. These can be evaluated according to the policy outputs (i.e., enforcement variables, change of perspective of street-level staff, etc.), policy outcomes (i.e., unemployment levels, life-expectancy of population, etc.) or indices of policy system change (i.e., administrative re-organization, privatization, etc.) (44). While the measures and levels will vary depending on the size, scale and focus of implementation, there is broad agreement that outcomes should be clearly defined a-priori and precisely measured. Evaluation findings regarding outputs and outcomes can dynamically feed back into the implementation process as it unfolds. This creates feedback loops and the process becomes very dynamic and multi-directional.
Part 2 – Determinants framework
Figure 3 presents an overview of our determinants framework and the relationship among the determinants. Our findings point to three sets of policy-related factors that affect the process, outputs and outcomes of implementation: 1) policy instruments and strategies; 2) determinants of implementation; and 3) policy actors, including their characteristics, relationships and context. Collectively, these feed into the process of implementation that proceeds in an iterative fashion along the stages: exploration, installation/preparation, initial implementation, full implementation/sustainment (12,41). The types of policy influences vary according to the stage of implementation (12). The process of implementation leads to a variety of outputs and outcomes as described above.
Policy instruments and strategies
Policy instruments and strategies are the most common set of factors mentioned in the literature and we found evidence for each of the instrument types described here, although with varying levels of detail. Policy instruments can be applied to implementation in differing ways, often with two or three levers used concurrently to implement a single initiative or strategy (47). In order to classify these strategies in a meaningful way, we drew on and adapted elements of a mutually exclusive and collectively exhaustive framework that identifies key features of health and social systems (39) and honed in on strategies that are particularly important for implementation (Table 3). These include strategies focused on the governance arrangements, financial arrangements, service delivery arrangements and implementation-related supports in systems. We then divided these strategies according to the intended “target” of implementation. Common targets of implementation from a policy perspective include the whole system, organizations, the workforce or service providers, consumers, and the innovation itself (the EIPP to be implemented). We wish to note, however, that because policy-related variables have not necessarily been treated with the same specificity as other types of implementation variables, the most common strategies do not reflect the full array of strategies that could be employed.
Our framework identifies eight categories of determinants (see “Determinants” box and elsewhere in Figure 3). Each of these categories represents a suite of factors that are hypothesized to independently affect implementation outcomes. These determinants are described briefly below and in more detail in Table 4.
I – Characteristics of the Evidence-Informed Policy or Practice (EIPP) - The success or failure of a particular policy package cannot be evaluated based on its intrinsic characteristics alone (44). Instead it is important to examine whether the policy selected is an appropriate “fit” with the problem (48), well-justified (49) and aligned with existing context (12,50).
II - Policy Formulation Process - This is the shape given to a policy by the initial formation processes (44). It includes who in government is responsible for formulating the policy, their legitimacy and the extent to which there is opportunity to provide feedback, how much feedback is given and the responsiveness in terms of adjustments made (44).
III – Vertical Public Administration and Thickness of Hierarchy - Vertical public administration is the term used to identify the layers in the policy transfer process. It refers to separate governments exercising their authority with relative autonomy (44). Policies generated outside of a socio-political level may be more or less acceptable to that level. Within a given layer, a particular policy area may require the mobilization of any number of institutions, departments or agencies and these agencies must act in a coordinated, interdependent fashion, termed “thickness of the hierarchy” (51).
IV – Networks/Inter-organizational Relationships - The existence and nature of the relationships between parallel organizations who must collaborative in order to achieve effective implementation and who do not have a hierarchical relationship (44).
V – Implementing Agency Responses - The factors affecting the responses of implementing agencies can be divided into issues related to the overall characteristics of the agencies and the behaviour of front-line or street-level staff (13,44).
VI – Attributes and Responses from Those Affected by EIPP- Attributes include the diversity of target group behaviour and the target group as a percentage of the population (14). Responses include thing like impacts on workforce stability (12).
VII – Timing/Sequencing - As implementation is a process that unfolds over time, it does not always align with the cycles to which it is subject and the time constraints inherent therein (52,53). Additionally, the external context in which implementation occurs is ever changing and “quintessentially unstable”, and success hinges on the ability to perceive those changes and take the necessary actions to adjust along the way (54). In Figure 3, Timing/Sequencing is placed outside of the Determinants box to reflect its importance across all of the other elements.
VIII – External Environment or Policy Context – Much of the literature identified factors outside the policy area of focus that may influence implementation (Figure 3, outside the hatched line). Many authors referred to this generally as the “political and social climate”, as unmodifiable or macro “context” or as “socio-economic conditions” (9,14,55–60). We organized this determinant using: 1) the 3I+E framework (61), and 2) a taxonomy of health and social system arrangements (62).
In general, these categories of determinants should be viewed as interactive and not completely discrete (44) and the inter-relationship among the determinants is key (37).
Our analysis revealed a wide range of policy actors who are important for implementation. In an attempt to create a category of variables that is analytically useful across contexts, we first divided the types of policy actors into the broad categories of: political actors, bureaucratic actors, special interests and experts (63). To provide more specificity, we further divided these into a non-exhaustive list of actor sub-types that were frequently mentioned in the literature and included examples of the types of roles they tend to assume in implementation (Table 5). While many of the sub-types are commonly identified in other phases of the policy cycle, some receive particular attention in the implementation literature. These include two types of Special Interests: 1) Implementing Agencies - organizations or programs that are responsible for implementing the EIPP (e.g., hospitals, schools, etc.); and 2) Street-Level Bureaucrats who, due to the relatively high degree of discretion in their jobs, and therefore discretion over the dispensation of public benefits or sanctions to citizens, can be critical to realizing any large-scale implementation efforts. There are also three Expert sub-types that are particularly visible during implementation: 1) Field or Practice Leaders who can be influential in supporting practice change amongst professionals; 2) Innovation Developers/Disseminators who have developed the EIPP to be implemented and who may contribute or adapt tools and other types of support to encourage successful implementation; and 3) Intermediaries/Technical Assistance Providers who are organizations, programs or individuals that work “in between” policymakers, funders, and frontline implementers, to facilitate effective implementation drawing on expertise in implementation.
There are also three categories of actor-related variables that are important: 1) actor characteristics; 2) actor relationships; and 3) the context in which the actors are embedded. (Figure 4). First, the characteristics of the policy actors (either individual- or organizational-level) such as their knowledge of the implementation context, their legitimacy, power and control, and their leadership in the context of the implementation effort are often cited as being critical to the success in large-scale implementation initiatives. Second, the relationships policy actors have with other actors, such as the level of shared values and beliefs or the coordination and alignment of actors and their activities, can be predictive of successful implementation. Finally, the context of the actors, such as the sustainment of political will and commitment and the stability of the actors themselves can predict the long-term success of implementation.