Five hundred and thirty-four survey links were sent via email, 88 viewed the survey and 54 experts participated in the Round 1 survey. The characteristics of participants are shown in TABLE 1. The majority of participants were women (59%) in the 55 to 64 age range (30%). The majority of participants (83%) were Researchers, Scientists or Professors living in Canada (41%), United States (39%) and the United Kingdom (7.4%). The majority of participants rated their experience with Implementation (83%), KT Science (72%), Dissemination (72%), KT practice (63%) and Integrated KT (61%) as high or expert. De-implementation expertise was rated as high or expert by 23 participants (43%) (See FIGURE 1).
Survey questions pertained to the content of the elements, order of the elements and comprehensiveness of the sub-domains. In Round 1 participants validated 46 of the 52 content questions. The questions which were not validated concerned the comprehensiveness of the Element (n=4) (Engaging Stakeholders, Monitoring and Evaluation) and the order of elements within a sub-domain (n=2) (Developing the Implementation and Sustainability Plan, Monitoring and Evaluation). Mean scores ranged from 3.4 to 4.4, with the standard deviation ranging from 1.2 to 0.5. TABLE 2 provides the results for Round 1, including the mean, standard deviation, median, IQR, percent agreement. Questions that reached consensus (greater than 75% agreement) were considered validated unless qualitative data was contrary and reached consensus to amend the item.
Participants also made recommendations in the comments sections regarding items for addition or removal. Content analysis of the recommendations and feedback provided in the Round 1 survey identified themes to be addressed and incorporated into the revisions for Round 2. The key themes identified in the Round 1 survey:
Participants emphasized that stakeholder engagement should not happen at a specific point in the process, but rather is critical throughout the planning, implementation, monitoring and evaluation processes. One Delphi participant commented: “There is a 'stream' of stakeholder engagement work that cuts across all domains. Some of the work has a natural sequence and some might be done by different or the same stakeholders at roughly the same time period.”
Participants noted that engagement should be integrated throughout and must also involve accountability and responsibility for all parties. A participant noted: “There should be something added around ensuring meaningful engagement of stakeholder partners e.g., through building trusting relationships, valuing diverse expertise and knowledge, shared decision-making, shared goals, etc.” To address the feedback received, the process model was refined, and Engaging Stakeholders was included throughout the three sub-domains.
The expert panel also felt that the importance of understanding, identifying and planning for the impact of context on the implementation process was underrepresented in the model. A number of participants stated that context and the actions which address it, need to be explicit in the process model: “I don't see how the issue of context is highlighted; it may be implicit, but in my view since implementation is a function of the intervention by context interaction, context and potential interactions should be explicit.” In response to this feedback, context and the actions required to address it were explicitly added to elements of the process model. Context was made explicit in the first step of planning: Identify the purpose of the Implementation and Sustainability of the intervention/innovation. In addition, context was incorporated into 4 other elements where it was applicable.
Implementation as an iterative process
Many participants discussed that implementation and sustainability are iterative non-linear processes. Participants acknowledged the need for logical presentation and helpful heuristics when documenting implementation in a process model but asked that the non-linearity of implementation be highlighted. A participant stated: “… you need to be clear that these are steps to be covered, not steps to be followed. Iteration will often be necessary, and flexibility is required depending on the situation.”
One participant emphasized the impact of non-linearity on implementation efforts: “Planning allows us to prepare for contingencies, to form alliances, to gather resources. It allows us to articulate a clear statement of our intentions, and of the actions needed to achieve those intentions. However, when the plan is complete and action has begun, it is essential that we do not follow a rote, fixed implementation of the plan. Rather, we watch the plan as it unfolds, we notice what is working or not working, and we revise and adjust as we go. Each situation will be different, each social form will be characterized by unique affordances and constraints. We are firm in our intentions and flexible in our actions.” The guidance for the process model was amended to explicitly acknowledge that implementation is an iterative process and that the elements detailed in the model represent evidence-based components to consider and address to support implementation, but do not require a sequential completion.
Use of theories or frameworks
The value of using theory or frameworks to guide implementation was also highlighted by participants. One participant commented: “One always uses a framework or mental model. The only question is whether it is made explicit. And it should be.” Participants also discussed the importance of aligning theories or frameworks with the intervention. One participant noted: “[This Element] should state that the framework must be matched to the problem and determinants.” The selection of guiding frameworks was moved up in the model and additional guidance was added regarding selection and application of theories and frameworks.
Amendments as a result of Round 1
Nineteen changes were made to the process model based on the responses received in Round 1. Changes applied to location (n=11), removal (n=4) and addition (n=4) of sub-domains/elements/sub-elements. These changes were reported in a table of substantive changes made to the process model and a refined version of the process model.
For Round 2, 59% of Round 1 participants completed the survey (n=32). The 19 amendments to the process model were represented in 23 survey questions which were evaluated by the participants. Again, participants were provided with comment sections on each question to provide additional feedback. All 23 questions were validated by the panel in the Round 2 survey. Participants provided additional feedback on the need for consistent terminology and also the need to further clarify the target user for the tool. Mean scores ranged from 3.8 to 4.8, with the standard deviation ranging from 1.0 to 0.3. TABLE 3 provides the results for the Round 2 Survey. FIGURE 2. illustrates the Delphi Process Summary. FIGURE 3. demonstrates the final, validated Implementation Process Model.