Six MIP projects opted to participate in interviews for Roadmap development, with five of these projects recognized as having shepherded in successful community-based implementation (Table 1). N=34 individuals representing these six projects were contacted for study interviews, and N=28 (82%) completed interviews. N=14 (50%) were PIs, Project Managers, and study staff of six MIP study teams. All six projects provided name(s) of community-based implementation partner(s). N=11 (39%) interviews were completed with key stakeholders at these sites, with at least one site represented for each of the six projects. N=3 (11%) interviews were conducted with MDHHS staff. N=23 (82%) interviews were completed in-person, while N=5 completed over the phone upon their requests (with three MDHHS staff and two MIP project staff) (Table 2).
Informing the Roadmap
Our analyses identified “best practices”, which we then mapped onto the three phases of the REP framework and used to identify actionable steps implementers could take to help anticipate, accommodate, or alleviate implementation barriers (Figure 1, Table 3). Within each implementation phase, we present the steps identified and results from analyses explaining how they advance implementation.
Generally, interviewees recognized that preparatory work, prior to any active implementation effort, is fundamental to successful implementation efforts. However, even among the successful MIP implementation teams, both study staff and community stakeholders recognized that more time and effort could have been used to prepare. Multiple MIP study teams recommended budgeting at least double the amount of time originally estimated for their preparation stage. To guide the pre-implementation phase, summarized in Table 4, we identified seven steps for implementers to take: (1) identifying high-priority needs; (2) engaging stakeholders; (3) selecting essential metrics; (4) assessing contextual factors; (5) clarifying core features of the EBI; (6) refining the EBI; and (7) adapting select features to local contexts.
Identify High-priority Needs
An early pre-implementation step for all MIP projects was the identification of needs and priorities within the communities they planned to serve. Several teams communicated that a mismatch between the needs of the community and the needs of the project could curtail implementation efforts, and encouraged aligning needs from very early on in the implementation process as a precursor to successful implementation.
Nearly every interviewee identified stakeholder engagement as crucial to ensuring successful EBI implementation. Especially in larger organizations, ensuring that all stakeholders were on board with the project from the beginning decreased the time spent on other pre-implementation steps. Leadership buy-in was cited repeatedly, but teams also stressed the importance of bottom-up engagement. Teams stressed the importance of regular, in-person or videoconference meetings with site stakeholders, ideally with the PI present, as a way to both increase stakeholder engagement and ensure sites that study teams were invested in the EBI’s success.
Select Essential Metrics
Study team members said early metric selection was important for measuring progress during the implementation process, but also helpful for engaging stakeholders during pre-implementation, and ensuring similar goals and expectations for the project across stakeholders. Project teams and investigators recognized the importance of selecting and measuring baseline values for key metrics prior to any implementation effort for informing future sustainability. For example, accurate baseline measures of provider use and fidelity of the EBI, as well as receipt, quality of care, and outcomes among consumers who received or did not receive the EBI are important to determine the impact of EBI implementation.
Assess Contextual Factors
In line with frameworks like the CFIR, assessing the context allowed study teams to identify potential barriers or facilitators to their proposed project. Typically, the contextual factors identified by teams were those defined by the CFIR as “inner setting” characteristics (18); for example, anticipated employee time/effort available to implement the EBI, availability of protected employee time to carry out EBI implementation, current or anticipated staffing shortfalls, physical space availability, technological capacity (e.g., Electronic Health Records), and/or internal site regulatory processes.
Clarify Core Features of the EBI
Several successful projects noted that once stakeholders were engaged and the EBI was deemed appropriate, it was important to ensure that all stakeholders understood the core features of the EBI prior to the beginning of active implementation. Study teams highlighted the importance of talking with the community stakeholders about both the scientific and practical components of the implementation effort, including outlining the project timeline and steps in the implementation process (e.g., opportunities for training, consumer follow-ups) as well as being very explicit about expectations for logistical needs, such as staffing, effort, and space. Explicitly clarifying features for all involved is important, especially when prioritizing efficiency, and necessary for completing MIP objectives in a limited time.
Refine the EBI
As is the case with most investigator-led implementation efforts, MIP teams all had an EBI in mind that they planned to implement--and indeed were funded to implement a specific EBI. Nonetheless, teams still reported that assessing the fit between their selected EBI and the identified high-priority need(s) of the site(s) was a key step. Successful projects noted that they remained open-minded about potential refinements of their intervention to ensure that implementation would be feasible in their community setting. Refining an intervention to fit the needs and setting, while ensuring preservation of essential evidence-based elements, is a key step in allowing implementation to move forward. Note that refinement of the EBI may take a couple of iterations, such as through the use of rapid-cycle testing of the EBI implementation, especially as additional barriers are discovered through assessment of contextual factors.
Adapt Select Features to Local Context
Core feature identification, EBI refinement, and contextual factor assessment all helped study teams to adapt their selected EBI to their local site’s context. Nearly all studies we spoke with mentioned that some kind of adaptation had to be done, but not all sites had planned for adaptation prior to active implementation. Among several documented ways to plan for local adaptation, one that emerged from our interviews was workflow analysis. The workflow analysis process, an essential way to understand the EBI core elements, as well as to identify adaptation or menu options to local context, (e.g. via rapid cycle testing), helped the study teams to lay out a step-by-step plan detailing how their project would be carried out with their community partners. This process proved especially helpful to MIP teams implementing their EBI in multiple community settings, as it allowed them to systematically consider between-site differences, as well as concomitant differences in adaptations necessary to accommodate the variations. Another way that many teams adapted their EBI was by tailoring projects to local context based on feedback from community sites.
Common practices to promote EBI implementation emerged across teams. From these commonalities, we identified six steps for the implementation phase (See Table 5): (1) soliciting feedback for further EBI tailoring; (2) communicating regularly; (3) maintaining engagement; (4) developing and utilizing resources; (5) supporting transitions; and (6) collecting and monitoring key metrics.
Solicit Feedback and Further Tailor the EBI
During active implementation, teams described seeking feedback from community sites and tailoring their program accordingly. Teams found that their method of implementation often needed to be tailored to particular site needs mid-course during the implementation effort, such as accommodating new barriers to implementation or adjusting to leadership changes. Although tailoring done in the pre-implementation phase was important, further tailoring during active implementation, including course corrections and re-tailoring, allowed teams to readily respond to unanticipated barriers and between-site differences.
Regular communication was important for bringing an EBI to a community setting and maintaining its implementation. Best regular communication practices were not limited to content, but also included thoughtfulness and efficient communication. Engaging in regular communication while being careful not to overburden partners is a fine line for teams to walk. Steps that reduce the communication burden on community sites, such as minimizing the number of reporting documents required or giving ample notice when reports are due, enhances effectiveness of communication.
Maintaining provider engagement with the EBI was frequently noted as one of the more challenging aspects of implementation. Sending reminders was one of the frequently cited tactic, while PI’s on-site presence and face-to-face meetings were described as effective ways to maintain engagement. When on-site time was not possible, teams reported maintaining site engagement through electronic tools and online spaces to share experiences and feedback. Others reported maintaining communication between sites in similar geographic areas so that on-the-ground implementers could share their experiences with the project and engage beyond the MIP project team, further creating space for engagement.
Develop and Utilize Resources
Improving resource utilization, for example through the development and adaptation of high-quality resources related to the EBI (training manuals, websites, etc.) that community sites could easily access, was an important implementation step for many teams. Technology development around project delivery was cited by research teams as another resource that aided their implementation work. Community partners viewed technology resources as exciting and useful tools for them, especially when the introduced technology was easy to access and user friendly.
Teams noted that having supports for participating staff’s transitions in the site is important step, as many of their community sites experienced high turnover both at the provider and leadership level. While reflecting on differences in implementation across sites, project teams shared working with sites with multiple participants were much more likely to allow projects to continue through transitions. Teams noted that with high turnover, and without a plan to introduce and transition the MIP project to the new staff member, the project was left behind without clear steps for onboarding new staff and study teams felt like they needed to start again at square one.
Collect and Monitor Ongoing Metrics
To both aid implementation and prepare for the sustainability, collecting and monitoring identified key metrics was helpful. This includes data on EBI use and quality, consumer outcomes, implementation fidelity, and provider engagement. In particular, metrics for fidelity were used to ensure MIP projects could be applied with fidelity in their chosen setting, which in turn was a key to overcome recognized implementation barriers and to support successful adoption of the EBI. Teams noted that, throughout the full implementation effort, they maintained various tools to ensure the fidelity of their implementation strategies, including developing a fidelity tool, selecting specific components to monitor, using self-ratings, and using observed ratings.
To sustain MIP projects beyond initial implementation efforts, MIP teams benefited from having a sustainability plan in place to guide them throughout the implementation process. While plans varied by teams based on funding sources, long-term goals, and other team-specific factors, many teams took similar steps to move their projects to a sustainable model. For the sustainability phase of the Roadmap, we identified four steps for implementers to take as summarized in Table 6: (1) analyzing and using collected data, (2) determining costs and establishing a return on investment, (3) establishing a business model, and (4) planning to transition ownership to stakeholders.
Analyze and Use Collected Data
The data collected by MIP research teams throughout their EBI implementation were the most useful tools they had to advocate for resources to help promote the sustainability of their projects. Teams utilized data to show stakeholders the impact of their programs. Analyzing collected data was also a benefit for the teams themselves by highlighting successes and positive outcomes. Sharing and presenting results from data collected throughout the implementation process helped the project teams to advocate for the sustainability of MIP projects while also allowing teams themselves to reflect on, show progress, and continue improving the implementation of their EBI.
Determine Costs and Establish Return-On-Investment
Replicating, scaling and maintaining EBI implementation requires a determination of costs. Questions around costs are often one of the first factors that site leadership (or policy makers) consider when evaluating projects’ impacts (27,28). This includes not just the costs to support the EBI directly, but also staffing time for practitioners delivering the EBI and program oversight, technological costs, physical resources, and costs for documenting any key metrics or other EBI information. Exact inputs will vary across programs, and implementers should take time to consider these inputs when determining their program costs.
An important step in moving towards sustainability is establishing the return-on-investment for community-implemented projects. The ability to demonstrate to stakeholders that the resources necessary to start and maintain the EBI can pay dividends, for example, improved clinical outcomes, increased community throughput, or decreased employee turnover or burnout, improves the potential for scaling EBI implementation efforts.
Establish Business Model
Implementation projects’ ultimate goal is to have a model where projects are not tied to grant funding and can sustain independently of short-term mechanisms (e.g., grant funding). For many MIP projects, the biggest barrier to sustainability was a lack of billing codes for the practice. MIP’s partnership with MDHHS provided access to key policymakers that could help implementers pursue changes to billing codes (29); however, metrics related to cost and effectiveness were necessary to motivate policy change. Establishing a business model for the EBI that includes information on the costs, return-on-investment, as well as the relationships needed to move a sustainable model forward furthers implementation efforts by showing how projects can exist outside of the MIP/Medicaid funding model. In many cases, however, the knowledge necessary to develop a strong business model is not native to the implementing team, so frontline implementers may want to consider contracting this expertise.
Plan to Transition Ownership to Stakeholders
One of the last steps in sustaining EBI implementation is transitioning ownership from the MIP research teams to the stakeholder(s) at the community organization where the EBI has been implemented. This includes preparing for ownership transition to stakeholders by developing plans with implementers at their site, identifying who would review outcomes and measures, who would be able to re-correct the project if necessary, and who would house and disseminate training tools when workforce turnover occurs. Pulling from the work done in other stages outlined in the Implementation Roadmap, research partners, community sites, and other stakeholders can work together to ensure a viable plan for ownership transition is in place in order to support the future of these projects.
Stakeholders at MDHHS are also invested in the sustainability of these programs beyond the Medicaid matched funding mechanism. Partners at MDHHS have said that they want projects that have a sustainable funding plan and utilizing these relationships to transition financial ownership is a key step in garnering longer-term sustainability for successful projects. For example, if pursuing billing codes for specific services, teams need to be linking up with the correct people from the beginning to pursue this funding route. MIP is uniquely situated in that these projects can connect with key personnel in the government of the State of Michigan because of the partnership with MDHHS. This mechanism may make it easier for options like approving to fund (or pay for) certain billing codes to be pursued through the relationships this partnership affords (29). Additionally, the State of Michigan has already demonstrated their support through its decision to fund these programs and may be more able to help sustain the delivery of these programs through alternative funding mechanisms.