Implementation stages for integrating behavioral health and primary care: A participatory Delphi study to refine a determinant framework

doi:10.21203/rs.3.rs-2325582/v1

Download PDF

Research

Implementation stages for integrating behavioral health and primary care: A participatory Delphi study to refine a determinant framework

https://doi.org/10.21203/rs.3.rs-2325582/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: The integration of behavioral health and primary care is a best practice to improve patient outcomes and achieve health equity. However, the process of integrating is complex, requiring organizational change and sometimes a complete system overhaul. Implementation science offers useful ideas for helping healthcare organizations to implement effective care, including recognition that change occurs incrementally and is stymied by specific barriers and facilitators. Although determinants (i.e., barriers and facilitators) of successful care integration have been identified their relevance to organizations seeking to integrate care remains unknown. This study sought to understand which implementation determinants (defined by the R=MC² framework, where Readiness is conceptualized as Motivation, Innovation-specific Capacity, and General Capacity) are most important in the process of integrating care across four implementation stages: Exploration, Installation, Initial Implementation, and Full Implementation.

Methods: A participatory Delphi study design was used to gather practice-based evidence from ten integrated care practitioners, researchers, and technical assistance providers. Qualitative and quantitative data were collected across four rounds. Qualitative summaries were presented back to participants between rounds for reaction and recalibration. The Framework Method guided initial qualitative analyses, which were then triangulated with descriptive statistics from a process survey. Coders used fuzzy set theory to create the final results on perceived relevance of each determinant across stages of implementation.

Results: Determinants most relevant across implementation stages are Leadership, Priority, a Champion, Supportive Climate, Culture, Innovativeness, and Staff Capacities. Determinant strength is most critical during active implementation stages.

Conclusions: The present study is one of only a few that apply implementation science frameworks to the integration of behavioral health and primary care. Results provide a structure for assessing determinants by phase and could be used to determine precision and timing of resource allocation to facilitate integration.

Integrated behavioral health and primary care is a key strategy for increasing behavioral health access and mitigating mental health disparities.
Many conditions and determinants impact successful implementation of integrated care, but it is unclear when and how to improve those determinants.
This study uses participatory methods to identify the most salient determinants for integrating care by implementation stage.
Results could be used to inform implementation guidelines, assessment tools, and technical assistance.

Integrated behavioral healthcare is the provision of clinical care where medical and behavioral health providers collaborate to improve patients’ biopsychosocial health.¹ There is a wealth of evidence demonstrating that collaborative and integrative care are effective in benefiting patients’ health and reducing patient cost.^{2,3,4,5,6,7,8,9} However, many organizational barriers impede successful care integration.^6,10,11 Calls have been made for this integration to be supported across levels of the healthcare system including by governments, insurance companies, and healthcare providers.¹² Unfortunately, there is little existing knowledge on how to successfully integrate care.^13,14 A multitude of barriers and facilitators for integrating care have been identified in the literature, including leadership, individual adopter characteristics, provider communication, a champion, seeing progress, provider education and knowledge, management support, innovation-values fit, culture, staff capacity/turnover, motivation, burden, socio-political climate, funding, organizational characteristics, and delivery structures.^6,15,16 Contextual factors; for example, practice type, like a federally qualified heath center versus a community mental health center – can also influence the level of coordination, co-location, or integration.¹⁷ Several methods have been proposed to address these barriers.^18,19 The current strategy to integrate care uses guides with checklists of activities to complete; while these are useful for structuring the process,^18,19,20 completing prescribed activities alone has been demonstrated to be insufficient.^6,10,11,21

The R = MC² framework^22,23 provides a comprehensive list of 19 implementation determinants and contextual factors that includes all the barriers and facilitators noted above and is generalizable for any setting or model of integrated care. R = MC² states that organizational readiness is an amalgam of three primary components: Motivation, Innovation-specific Capacity, and General Capacity. These components are present to varying degrees within organizations and can change over time. Motivation refers to the “perceived incentives and disincentives that contribute to the desirability to use an innovation.”²² This includes the organization’s collective beliefs about the innovation and the degree of support for making the change. Motivation is specific to the innovation. Innovation-specific Capacity refers to the human, technical, and fiscal conditions for implementing a particular innovation.²⁴ This includes the knowledge and skills necessary for implementing an innovation, the climate of support, and whether there is an influential person signaling support for it within the organization.²² General Capacity refers to attributes affecting overall functioning of the organization and is untethered to a specific innovation. This includes leadership strength, resource allocation, culture, and staff capacity, among other subcomponents.²² The original framework was expanded to include additional subcomponents and refine the initial labels and definitions,²⁵ and is collectively a list of implementation determinants derived from both literature and practice.

Despite roots in the well-established Interactive Systems Framework for Dissemination and Implementation (ISF),²² R = MC² and its parent framework (ISF) are limited. R = MC² has been used to assess care integration;^23,26 yet, it is unclear how the determinants can be actionable for helping clinics to integrate. Importantly, although evidence shows determinant salience varies over time,²⁷ the framework lacks consideration of how these determinants may be differentially important across implementation stages. This presents a feasibility issue: it is impractical to suggest a healthcare clinic or practice continually assess and improve 19 implementation determinants over the several years required to integrate care.

Achieving successful implementation of integrated care requires expanding R = MC² to understand how its determinants vary by implementation stage. Across implementation frameworks, there are roughly four stages: Exploration, Installation/Preparation, Initial Implementation, and Full Implementation or Sustainment.^28,29 Each stage prescribes implementation activities where strategies may be enacted for smoothly transitioning the organization from existing practices to effectively sustaining an innovation. Exploration is the first stage, where the organization assesses their needs and resources to determine what kind of innovation may be helpful. Installation involves preparing the necessary resources and supports for the innovation. Initial Implementation executes organizational changes necessary for the innovation to be implemented. Full Implementation is reached “when most practitioners are routinely providing the new or refined program model with good fidelity.”²⁸ This requires functional implementation determinants that were improved or modified in earlier stages to scale and sustain the innovation. While other determinant frameworks already consider implementation stages (e.g., Active Implementation Frameworks²⁸), these have not been applied to integrating behavioral health and primary care. R = MC² was selected because it is an implementation framework with documented application and adaptation to the integrated care field.^23,26

Many conditions and determinants impact care integration, but it is unclear how to improve them in practice. In addition, some determinants may be more important than others, or only important at certain times during the implementation process. The process of implementation is lengthy and occurs in different stages. This is acknowledged in most implementation frameworks, yet the relevance of an organization’s current stage of change is rarely considered in practice. This study seeks to address gaps in research and practice by further specifying the relevant determinants of implementation success for healthcare organizations to integrate behavioral health and primary care. Employing participatory methods and expert feedback, the study aims are to understand which implementation determinants are most important for successful integration at each stage of implementation and propose how these findings could be incorporated into a practical implementation support tool. Findings may provide a template for the timely allocation of strategies to smoothly facilitate integrating care.

Method.

The Delphi method is a pragmatic process of community-engaged inquiry used to gain insight into a complex problem.³⁰ This method systematically collects the opinions and lived experiences of content experts to create a collective judgement on the topic in question, often for the sake of informing policy and practice.^30,31 Experts include both researchers and practitioners. The aim is to reach group consensus on the problem, accomplished through iterative rounds of inquiry and confidential feedback by the respondents.³² As a participatory action research technique, the Delphi method involves participants at every stage of the process: survey design, interaction with others, interpretation of results, and application of results.^33,34 It must also serve the needs of not just the researcher but the participants as well, where participants generate solutions and additional action items.³³ This includes privileging participant opinions to design the study, translate the results, and inform policy.³⁴ Supplement A provides more methodological details.

Recruitment Procedures.

Recruitment took place in August 2018 and data were collected from August to December 2018. Determination was made a priori to hold three rounds of primary data collection: individual semi-structured interviews (Round 1), and two structured online surveys (Rounds 2 and 3). Consistent with Delphi practices, the procedure was iterative with each round informed by the previous. The first three rounds were conducted independently to ensure anonymity; only the researcher knew the identities of panelists. The study culminated in a debrief with all participants. Participants were invited to help co-design and/or pilot each round of the study; no one elected to do so. However, procedural suggestions made by participants during the Delphi study process did inform each round.

A maximum of 15 participants were sought for recruitment based on the suggestion that 8 to 15 panelists is ideal for a Delphi study.³⁵ Because all participants must move through the process collectively, rolling recruitment is not possible. To leave room for attrition, it was determined that a minimum of 10 participants would be necessary to begin the study. Using purposeful sampling,³⁶ individuals were invited to the study based on study staff knowledge that they would meet experiential criteria (described below); additional participants were recommended by the initial pool of potential participants. Expertise worthy of inclusion on the panel was determined by peer nomination (i.e., researchers and practitioners identified by the author nominated other researchers and practitioners as potential participants), screening questions for self-reported experience, and additional inquiry on details of experience during the Round 1 interview.

Three narrow criteria were used to nominate participants, with the expectation that each participant must meet at least one: Criterion A) Staff (e.g. physicians, nurses, behavioral health specialists, administrators) currently or previously (for at least one year) at a clinic/practice that has integrated behavioral health and primary care where previously they did not have an integrated system; Criterion B) Consultants or technical assistance providers for clinics integrating care, who had provided this type of support to clinics for at least one year; Criterion C) Researchers who have studied both implementation science and integrated care for at least one year. These criteria were selected to capture the three different systemic levels of influence for implementation success within the framework that provided the conceptual basis upon which the R = MC² determinants were built;²² in the Interactive Systems Framework (ISF)³⁷ this represents actors within the delivery system (Criterion A), support system (Criterion B), and synthesis and translation system (Criterion C). Multiple criteria ensured a diversity of experience on the study panel, consistent with Delphi method best practices for dependability.

Data Collection Procedures.

Qualitative and quantitative data were simultaneously collected throughout the procedure. There were two aspects of data collection and analysis: primary data collection with iterative analysis (Delphi study process) and secondary data collection (post-debrief process evaluation survey) with post-study analysis. Interim analytic processes variably prioritized qualitative then quantitative data by round, with both given equal weight for determining final study results (QUAL + QUAN³⁸).

The first round was conducted as a one-on-one phone interview. This interview obtained background information on the participant and confirmed their eligibility for inclusion, gathered initial opinions on the determinant framework’s relevance for integrated care, and weighed in on when during the process of implementing integrated care the determinants seem most relevant for success. Participants were sent an information sheet one week in advance of the interview. The sheet contained information about the study, the R = MC² determinant framework’s components and subcomponents, and information on two existing stage frameworks (one derived from implementation science, one derived from the integrated care literature). Qualitative data from Round 1 provided the basis for Round 2. Based on data from Round 1, the implementation science stages were deemed more appropriate for the study. The surveys for Rounds 2 and 3 were structured into three parts by the determinant framework’s components, with specific questions for each subcomponent. Within each item, the subcomponent was defined in terms of integrating behavioral health and primary care. Then a selection of panelists’ anonymized comments from the previous round were presented. Questions for each item included asking how important each subcomponent is for integrated care during each implementation stage (Exploration, Installation, Initial Implementation, and Full Implementation), with responses on a 7-point Likert scale from “totally unimportant” to “very important.” A comment box encouraged describing the rationale for their choice. The study culminated in a debrief, which served several purposes. First, the participants were provided with a preliminary results document; this transparency aids confirmability of the study’s rigor.³⁹ The document was created using the quantified consensus results from Round 3 in two tables, along with an overview of the study purpose, process, research questions, and potential implications. Second, the debrief provided panelists a chance to discuss openly their reactions to the results and the process and generate potential implications. Third, the debrief allowed panelists to introduce themselves to each other and openly debate their opinions on the topic. Their identities, experience level, and professional affiliations had previously been kept confidential. Fourth, as part of the reflection panelists could provide suggestions on next steps for the data. Interviews were recorded and transcribed. As a source of secondary data, panelists were invited to complete a process evaluation survey after the debrief. This was included as an additional form of member checks to ensure trustworthiness of the results.⁴⁰

Analyses.

There were three components of data analysis: process analysis, initial data analysis, and exploratory data analysis.

In initial analyses, the qualitative data were deductively analyzed using the Framework Method as a guide for coding and interpretation.⁴¹ After familiarization with recordings and transcripts, first cycle coding⁴² was conducted via NVIVO with a priori deductive codes. For second cycle coding,⁴² data sections on validity and implications were organized into framework matrices by participant (rows) and first cycle codes (columns). Pattern coding⁴² was employed to interpret themes in validity-related perceptions, study implications, and additional analyses suggested by participants. Pattern coding is a second-cycle coding process and is distinct from pattern matching (a qualitative method of deriving causal relationships). Additionally, to assess validity, descriptive statistics from the process evaluation surveys were converged with qualitative data.

Initial analyses identified concern from participants that preliminary results were quantified only by items being rated on the highest end of the Likert scale (“important” or “very important”). Participants suggested addressing this by considering the results by the distribution of scores rather than solely by percent agreement to capture whether ratings were also rated as “unimportant.” This suggestion led to the exploratory analysis.

Exploratory analysis considered results by distribution of scores, rather than solely by consensus percentages. Broadening the data interpretation shows strength of agreement in a different way. This indicates if respondents perceived determinants as wholly unimportant, or just less important compared to other determinants by stage. Percent agreement is the most commonly applied metric of consensus in Delphi studies,⁴³ which is why it was chosen for the preliminary analyses in Phase I. However, based on the participant suggestion of considering results in different ways, the Delphi literature was revisited. Agreement and internal reliability are valid metrics, in addition to consensus, for assessing Delphi results.³⁵ In Delphi studies, descriptive statistics and graphical displays of data are appropriate for assessing internal validity.³⁵ However, levels of dispersion (e.g., inter-quartile range, standard deviation) are not appropriate for ordinal Likert scale data^32,35,44 thus cannot be applied here. Variability (e.g., minimum, maximum, range), central tendency (e.g., median, mode), and frequency distributions (e.g., percentages) are alternative options for analyzing ordinal data. Therefore, final study results are gleaned from a matrix display of percent agreement on Likert scale ratings, range of ratings, and a frequency distribution (i.e., histogram).

Descriptive statistics are not generally subject to systematic interpretive methods. However, to prevent bias and inconsistencies in translating the quantitative results into practice implications, the study employed an iterative coding process to make inferences from the matrix display.⁴² Two coders first collaboratively determined the method for extracting relevancy (Table 1) from the descriptive statistics and visual display, then independently categorized each determinant, and finally reviewed their category labels and resolved discrepancies. Relevancy was defined as: “how important this determinant seems at this stage.” Using fuzzy set theory (specifically type 2 fuzzy sets⁴⁵), five relevancy categories were defined (Table 1): highly relevant, relevant, less relevant, irrelevant, variable relevancy/more information needed. Coding benchmarks were constructed for each relevancy category. To retain the integrity of participants’ perceptions rather than the coders’ perceptions, both coders were blinded to the items within the matrix and applied the relevancy category labels based solely on the coding benchmark and definition. After labels were applied for all items, category labels were compared and resolved via discussion to ensure consistent interpretation across items.

Table 1

Interpretative coding definitions
Category	Benchmark	Definition
Highly relevant	Clear consensus (75–100% of participants rated 6–7 on Likert scale); narrow range; high median; left-skewed distribution.	Participants indicated that this determinant is very important for implementation success at this stage.
Relevant	More consensus (50–74% of participants rated 6–7 on Likert scale); narrow range; high median; distribution somewhat left-skewed.	Participants indicated that this determinant is important for implementation success at this stage.
Less relevant	Less consensus (0–49% of participants rated 6–7 on Likert scale); and/or medium-wide range; and/or distribution somewhat normal.	Participants indicated that this determinant is less important for implementation success at this stage.
Irrelevant	Right-skewed distribution; and/or median value rated 1, 2, or 3 on a 7-pt Likert scale; narrow range	Participants indicated that this determinant is not important for implementation success at this stage.
Variable relevancy, more information needed	Unclear due to wide range, abnormal distribution, or high discrepancies between consensus percentage and other benchmarks.	Participants did not clearly indicate whether this determinant is important for implementation success at this stage.

Participants.

Ten participants consented to the study. Hailing from different states across the USA, most participants reported multiple types of experiences with integrating care. Participants were categorized by the three different experience types per both self-rating and a detailed description of their experience. Table 2 displays participant characteristics and study completion. Eight participants completed all study activities, including the interview (Round 1), the two online surveys (Rounds 2 and 3), and the debrief.

Table 2

Participant characteristics
	ID	1	2	3	4	5	6	7	8	9	10
Participant Information	Gender	F	F	M	M	F	F	F	M	F	M
	Degree	MD	PhD	MA	LCSW	MA, MPH	PhD	MA	MD	MD	PhD
	Location (State)	SC	WI	MI	NY	GA	GA	SC	CO	CO	PA
	Institution Type	Medical Center (Hospital System)	Medical Center (Hospital System)	Community Health Center	Community Health Center	Research Institute (University affiliated)	Research Institute (University affiliated), and Community Mental Health Clinic	Research Institute (Independent), and Community Mental Health Clinic	Research Institute (University affiliated)	Research Institute (University affiliated) and Medical Center (Hospital System)	Research Institute (Independent)
Experience Type	Practitioner	BH	BH	BH	BH		BH	✼	PC	PC	✼
	TA Provider		Y	Y	Y	Y	Y	Y	Y
	Researcher						Y	Y	Y	Y	Y
Study Completion	Round 1	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
	Round 2	✓	✓	✓	✓	✓		✓	✓	✓	✓
	Round 3	✓	✓	✓		✓		✓	✓	✓	✓
	Debrief	✓	✓	✓		✓		✓	✓	✓	✓
BH = Behavioral Health (psychiatrist or psychologist); PC = Primary Care (MD); TA = Technical Assistance ^✼Behavioral health practitioners with no experience practicing in integrated care settings

Perceived Validity.

Debrief discussions revealed that participants confirmed they saw the study as valid; five participants explicitly stated this. For example, Participant 3 said “A lot of [the results] make sense and seem on-par with how I conceptualize these things.” This general perception was extended to specific findings such as the importance of leadership across all stages. Participant 8 noted “I am delighted and want to celebrate [the leadership result]. I think that's exactly right. I think that's dead on and you just can't overstate how critical prepared leadership is at integrating care.”

Potential Implications.

In the anonymous and voluntary post-study process evaluation, five of the six respondents (83%) agreed or strongly agreed that the results of the panel had implications for integrating behavioral health and primary care; one responded “neutral” to this question. During the debrief, participants reported perceiving study results as helpful for assessing an organization’s readiness for integrating care and as a tool for technical assistance providers to assist an organization integrating care. Participant 8 noted that, as an assessment, it could be used by funders and policymakers to determine whether an organization is ready to innovate:

…being ready to take the integration of primary care and behavioral health on is such a vital and useful step to formalize… [and for small and large organizations] to just recognize how important an assessment of their readiness to take this on is, [because] the practice is at a capacity at a particular point in time will be different a year from that time. …[this is] a chance for a little policy and implementation compassion. It would be a compassionate act to say to a practice, ‘If you use this tool... You may very well be able to identify and advance where you're going to need some help, and you may also be able to decide that this is not something for you.’ [to save them] the heartache and the challenges and the disruption that [starting something they’re not ready for] is going to bring; That's a service. That is a good thing.

In contrast, Participant 7 offered a different perspective of assessment as a way to assure organizations that fluctuations in determinant strength are expected. She argues it would help:

“…to normalize for people where they are. …it provides some more evidence. Like, ‘Hey, it's okay that your innovation-specific scores are lower at the Exploration phase…. No one expects you to have high scores, and it doesn't mean that you're not going to still do well at implementing integrated care.’"

Similarly, as an implementation support tool Participant 2 said “it could be used for a roadmap for… organizations and practice facilitators to lay out what components are not important at different stages.”

Final Results

Results (Table 3; N = 8 interpreted by two coders) indicated almost all aspects of Motivation, Innovation-specific Capacities, and General Capacities are important for success at various times. Cells in Table 3 are color-coded to visually indicate potential relevancy in each of the four implementation stages for ensuring success integrating care.

Table 3. Determinant relevancy results

Before implementation activities begin (Exploration stage), two aspects of Motivation (perceived Relative Advantage of integrated care, and perceived Priority of integrating) are highly important, as is one aspect of Innovation-specific capacity (integrated care Champion) and three contextual factors that are not specific to integrating care (Leadership quality, organizational Culture, and the organization’s degree of Innovativeness). In addition, three Motivation determinants are relevant (perceived Compatibility of integrated care with the organization, and Ability to Pilot care integration activities); as are three aspects of innovation-specific capacity (a Supportive Climate for integrating care, Inter-organizational Relationships between organizations that support integrating care, and Intra-organizational Relationships within the organization that support integrating care); and two contextual factors (general Staff Capacities for staff performing their jobs well, and Process Capacities of the organization to plan, implement, and evaluate their activities and processes). Three contextual factors were cited as having variable relevancy (general organizational Climate, Resource Utilization in terms of how the organization acquires and allocates resources in general, and Internal Operations which refers to the effectiveness of the organization at communication and teamwork). Variable relevancy suggests unclear or highly inconsistent ratings for this item. Inconsistency may mean that the determinant’s importance could vary by context, integrated care model, or be in need of further study. Determinants salient for Exploration are likely important to lay groundwork for making initial decisions and building support for integrating behavioral health and primary care.

Overall, more determinants are relevant at Installation, Initial Implementation, and Full Implementation; nine determinants were rated as highly relevant across each of these stages (however, not the same nine in each stage), compared to six highly relevant at Exploration. However, there are some that emerge as important across all stages. Both Leadership and perceived Priority are highly relevant across all stages. Champion, Supportive Climate, Culture, Innovativeness, and Staff Capacities are highly relevant at three stages and relevant at one. Relative Advantage and Process Capacities are highly relevant at two stages and relevant at two. Compatibility and Intra-organizational Relationships are relevant at four. Only one determinant appears less relevant across stages: Simplicity, which is defined as “how simple it seems to integrate behavioral health and primary care.” Notably, during the debrief this determinant was also cited by participants as either irrelevant for integrated care or confusing to answer. Few determinants were rated as variably relevant at any stage; these five instances appear solely during Exploration and Installation stages.

Integrated care initiatives are complex and demanding and require reflexive inquiry to understand the dynamic processes affecting organizational outcomes when integrating care.¹⁰ Implementation science proposes that improvement is non-linear and requires continuous effort.^10,46 The field recognizes that contexts change over time and there is a need for formative evaluation and an appreciation for the contextual, structural, and motivational barriers than affect progress. These ideas are supported by the few authors who have applied implementation science frameworks to integrated care.^{6,46,47,48,49,50}

Study results reveal the variable salience of 19 implementation determinants for integrating behavioral health and primary care, conceptualized across four stages of change. In Supplement B we briefly contrast the findings of salient determinants by what is currently known about each determinant’s definition, related R = MC² determinants, relevance for integrated care, and (if available in the literature) hypothesized importance across implementation stages.

The conceptualization of implementation stages presented here differs dramatically from established levels of integrated care, which generally imply a progression from minimal collaboration to co-located care to fully transformed collaborative care.⁵¹ The problem with current integrated care levels is the implication that integration is an end to itself. Use of integrated care levels is reductive; while a useful heuristic, an idealized status for all healthcare clinics that may not be feasible nor necessary for the specific context. For example, in the Center for Integrated Health Solutions’ proposed levels,⁵¹ a level 6 is presented as the idealized state where there is full collaboration in an integrated practice, seamless patient transition, shared resources, a single treatment plan, and full provider buy-in. Yet, this may not be desirable in practices where separate resources or treatment plans work better. While there is evidence that integrated care improves patient outcomes and cost, there is no research evidence underlying the assumption that a level 6 integrated care practice is superior to other levels. As such, in this study participants were given the option between implementation stages⁵² and integrated care levels⁵¹ for conceptualizing the study, and overall chose the implementation stages as the best option. Given that implementation stages are generalizable to different innovations and practices the present study results can be applied across the continuum of care.

Many different models of collaborative care have been proposed. Some define integration in terms of the functional, organizational, or clinical differences,^1,20 others by the degree of coordination.⁵³ There are almost as many models for integrated care as there are healthcare organizations in the USA.⁵⁴ The conceptualization of implementation stages posed here describes a process that could be applied to any integrative care model. This has implications for the development of a generalizable assessment tool for practices seeking to integrate care. Further, with ongoing study of determinants’ amenability to change⁵⁵ and validated assessments of drivers for collaborative care,⁵⁶ a toolkit could be developed that specifies when to leverage resources for building implementation motivation and capacity. Such a toolkit could provide a critical support to developing the workforce to meet the growing demand for integrated care services.⁵⁷

Generalizability beyond integrated care

Study findings revealed more determinants are relevant at Installation and Initial Implementation compared to the other stages. This is logical because the most intensive implementation activities take place at these stages. Specifically, Installation requires building infrastructure and establishing supports and Initial Implementation begins the service delivery and quality improvement activities.⁵² Given that implementation stages are meant to be broadly applicable across innovations and contexts, it raises the question of whether the present results are specific to integrated care settings or may be generalizable to other innovations and contexts.

One published study to date has applied both implementation stages (as defined here) and R = MC² determinants in another context. Domlyn and Wandersman²⁷ conducted a Delphi study in a nationwide community coalition transformation program. Similar to integrated care, coalition transformation is a multi-faceted, complex process. Like the 2019 study, the present study found Leadership, Staff Capacity, Process Capacity, and Champion are consistently important across stages in both contexts. However, the present study concludes Priority and Culture are highly salient in integrated care, whereas these reached weaker consensus among community coalitions. Additionally, unlike in the present study, perceived Simplicity (previously labeled Complexity) did achieve relevance across several stages for community coalition transformation.²⁷

Another previous study assessed perceived determinants across implementation stages in youth-serving organizations using different frameworks for both the determinants and stages.⁵⁸ This 2018 study found far more determinants relevant during pre-implementation (roughly congruent with Exploration) than in the present study. There were several differences. For example, the fit between practice and organizational mission is similar to the R = MC² determinant Compatibility. The 2018 study showed agency leadership were most concerned about this during pre-implementation and sustainment (roughly congruent with Full Implementation), but not during active implementation (roughly congruent with Installation and Initial Implementation). In contrast, the present study found Compatibility relevant across all stages. As another example, support for sources external to the agency (in R = MC² this is Inter-organizational Relationships) were also noted in pre-implementation and sustainment, but not during active implementation.⁵⁸ The present study found Inter-organizational Relationships are relevant in Exploration, Installation, and Initial Implementation, but less relevant during Full Implementation. While multiple methodological differences are apparent between this 2018 study and the present study (differing frameworks, data collection practices, and organizational settings), this may suggest that the present study’s findings about which determinants are relevant in an implementation stage may not be generalizable to other settings. It is worth exploring further to understand if certain determinants are consistently salient by stage, independent of context.

Strengths and Limitations

Several strengths of the present study are evident. This study executed a rigorous, participatory approach to collect data. A comprehensive literature review was conducted to ensure that methods utilized best practices for ensuring credibility, dependability, and confirmability. This was accomplished by using a heterogenous sample representing different stakeholder roles, conducting iterative member checks through feedback rounds, and maintaining detailed records of the process for data collection and analysis. Additionally, participants reported finding the results valid. Finally, the participant pools for both phases included a range of professionals with disparate types of expertise. Multiple stakeholder groups were represented: mental health practitioners, medical practitioners, implementation support practitioners, healthcare administrators, and researchers. Additionally, participants hailed from locations around the USA and were employed in many different organization types. This enhances the potential generalizability of study results for different healthcare settings.

Multiple study limitations are notable. First, while the study did adhere to published best practices on qualitative Delphi study size, the participant pool was smaller than those seen in quantitative Delphi studies. Although the participants represented a diverse sample of experts, the sample size may raise questions of generalizability of findings. Another data collection limitation is that the determinants were presented as discrete constructs despite that they are likely connected and interdependent. This choice was made to adhere to the foundational framework. Across the 76 relevancy results (Table 3), five did not reach consensus for unclear reasons; this could be due to natural variability of the determinants’ importance by context, methodological issues related to interpreting that determinant, or pure chance.

Integrating behavioral health and primary care is critical for achieving public health goals for the 21st century, ensuring people obtain necessary mental health interventions, and maximizing service outcomes. To aid care integration, this study used a participatory approach to understand the differential importance of implementation determinants over time while integrating care. The information gleaned from this study can inform evaluation activities, coaching strategies, and/or measure development. The purpose of this work is to move the needle on creating practical tools for helping organizations to integrate behavioral health and primary care quickly and efficiently. Understanding when each area of motivation and capacity is most important for successful integration informs next steps for strategically integrating behavioral health and primary care.

ISF

Interactive Systems Framework for Dissemination and Implementation

R = MC²

Readiness = Motivation x General Capacity x Innovation-specific Capacity

Ethics approval and consent to participate

This study was submitted to the Institutional Review Board of the University of South Carolina and was deemed exempt from review. Full study information was provided as part of recruitment and participants provided verbal consent to participate.

Consent for publication

Not applicable

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was partially supported by a SPARC Graduate Research Grant from the Office of the Vice President for Research at the University of South Carolina.

Authors' contributions

AD conceptualized the study, conducted data collection and analysis, and drafted most of the manuscript. AW provided design support and editorial feedback. LP advised on conceptual design and methodological considerations and edited the final manuscript. NZ advised on data analysis and significant edits to manuscript structure. MW provided conceptual feedback and provided major contributions to the manuscript. All authors read and approved the final manuscript.

Acknowledgements

The following individuals were instrumental for conducting this study: Gigi Bastien, Giorgio Chatelain, Bryan Dovichi, Eve Fields, Stephanie Gold, Larry Green, Julie Kaprelian, Virna Little, Sharon Rachel, and Michael Schoenbaum. For their assistance in various components of this project, additional thanks are extended to Victoria Chien Scott, Tara Kenworthy, Jenny Kolodny-Goetz, Mariajosé Paton, Jonathan Scaccia, and Amber Watson.

Muse AR, Lamson AL, Didericksen KW, Hodgson JL. A systematic review of evaluation research in integrated behavioral health care: Operational and financial characteristics. Families, Systems, & Health. 2017 Jun;35(2):136.
Camacho EM, Davies LM, Hann M, Small N, Bower P, Chew-Graham C, Baguely C, Gask L, Dickens CM, Lovell K, Waheed W. Long-term clinical and cost-effectiveness of collaborative care (versus usual care) for people with mental–physical multimorbidity: cluster-randomised trial. The British Journal of Psychiatry. 2018 Aug;213(2):456 – 63.
Camacho EM, Ntais D, Coventry P, Bower P, Lovell K, Chew-Graham C, Baguley C, Gask L, Dickens C, Davies LM. Long-term cost-effectiveness of collaborative care (vs usual care) for people with depression and comorbid diabetes or cardiovascular disease: a Markov model informed by the COINCIDE randomised controlled trial. BMJ open. 2016 Oct 1;6(10):e012514.
Green C, Richards DA, Hill JJ, Gask L, Lovell K, Chew-Graham C, Bower P, Cape J, Pilling S, Araya R, Kessler D. Cost-effectiveness of collaborative care for depression in UK primary care: economic evaluation of a randomised controlled trial (CADET). PLoS One. 2014 Aug 14;9(8):e104225.
Katon W, Robinson P, Von Korff M, Lin E, Bush T, Ludman E, Simon G, Walker E. A multifaceted intervention to improve treatment of depression in primary care. Archives of general psychiatry. 1996 Oct 1;53(10):924 – 32.
Padwa H, Teruya C, Tran E, Lovinger K, Antonini VP, Overholt C, Urada D. The implementation of integrated behavioral health protocols in primary care settings in project care. Journal of Substance Abuse Treatment. 2016 Mar 1;62:74–83.
Unützer J, Katon W, Callahan CM, Williams JW Jr, Hunkeler E, Harpole L, Hoffing M, Della Penna RD, Noël PH, Lin EH, Areán PA. Collaborative care management of late-life depression in the primary care setting: a randomized controlled trial. Jama. 2002 Dec 11;288(22):2836-45.
Unützer J, Katon WJ, Fan MY, Schoenbaum MC, Lin EH, Della Penna RD, Powers D. Long-term cost effects of collaborative care for late-life depression. Am J Manag Care. 2008 Feb;14(2):95.
Unützer J, Chan YF, Hafer E, Knaster J, Shields A, Powers D, Veith RC. Quality improvement with pay-for-performance incentives in integrated behavioral health care. Am J public health. 2012 Jun;102(6):e41-5.
Goodwin N. Improving integrated care: can implementation science unlock the ‘black box’of complexities?. International journal of integrated care. 2019 Jul;19(3).
Gold R, Bunce A, Cowburn S, Davis JV, Nelson JC, Nelson CA, Hicks E, Cohen DJ, Horberg MA, Melgar G, Dearing JW. Does increased implementation support improve community clinics’ guideline-concordant care? Results of a mixed methods, pragmatic comparative effectiveness trial. Implementation Science. 2019 Dec;14(1):1–4.
Crowley RA, Kirschner N. Health and Public Policy Committee of the American College of Physicians*. The integration of care for mental health, substance abuse, and other behavioral health conditions into primary care: executive summary of an American College of Physicians position paper. Annals of Internal Medicine. 2015 Aug 18;163(4):298-9.
Martínez-González NA, Berchtold P, Ullman K, Busato A, Egger M. Integrated care programmes for adults with chronic conditions: a meta-review. International Journal for Quality in Health Care. 2014 Oct 1;26(5):561 – 70.
California Mental Health and Substance Use Needs Assessment, January 30, 2012. Technical Assistance Collaborative/Human Services Research Institutes; 2012. Accessed November 15, 2022. http://www.dhcs.ca.gov/Documents/All%20chapters%20final%201-31-12.pdf.
Busetto L, Luijkx K, Calciolari S, Ortiz LG, Vrijhoef HJ. Barriers and facilitators to workforce changes in integrated care. International Journal of Integrated Care. 2018 Apr;18(2).
Moise N, Shah RN, Essock S, Jones A, Carruthers J, Handley MA, Peccoralo L, Sederer L. Sustainability of collaborative care management for depression in primary care settings with academic affiliations across New York State. Implementation Science. 2018 Dec;13(1):1–7.
Cohen DJ, Balasubramanian BA, Davis M, Hall J, Gunn R, Stange KC, Green LA, Miller WL, Crabtree BF, England MJ, Clark K. Understanding care integration from the ground up: five organizing constructs that shape integrated practices. The Journal of the American Board of Family Medicine. 2015 Sep 1;28(Supplement 1):S7-20.
Cash-Gibson L, Tigova O, Alonso A, Binkley G, Rosenmöller M. Project INTEGRATE: Developing a framework to guide design, implementation and evaluation of people-centred integrated care processes. International Journal of Integrated Care. 2019 Jan;19(1).
Ratzliff A, Phillips KE, Sugarman JR, Unützer J, Wagner EH. Practical approaches for achieving integrated behavioral health care in primary care settings. American Journal of Medical Quality. 2017 Mar;32(2):117 – 21.
Minkman M, Vermeulen RP, Ahaus KT, Huijsman R. The implementation of integrated care: the empirical validation of the Development Model for Integrated Care. BMC health services research. 2011 Dec;11(1):1 – 0.
Serrano N, Cordes C, Cubic B, Daub S. The state and future of the primary care behavioral health model of service delivery workforce. Journal of Clinical Psychology in Medical Settings. 2018 Jun;25(2):157 – 68.
Scaccia JP, Cook BS, Lamont A, Wandersman A, Castellow J, Katz J, Beidas RS. A practical implementation science heuristic for organizational readiness: R = MC2. Journal of community psychology. 2015 Apr;43(4):484–501.
Scott VC, Kenworthy T, Godly-Reynolds E, Bastien G, Scaccia J, McMickens C, Rachel S, Cooper S, Wrenn G, Wandersman A. The Readiness for Integrated Care Questionnaire (RICQ): An instrument to assess readiness to integrate behavioral health and primary care. American Journal of Orthopsychiatry. 2017;87(5):520.
Flaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Unpacking prevention capacity: An intersection of research-to-practice models and community-centered models. American journal of community psychology. 2008 Jun;41(3):182 – 96.
Readiness Briefing Paper FY 2018–2019. Wandersman Center; 2020. Accessed August 1, 2020. https://www.wandersmancenter.org/learn-more-about-readiness.
Domlyn AM, Kenworthy T, Scaccia JP, Scott VS. Organizational conditions that change over time: Assessment of readiness. In: Russ-Eft DF, Sleezer CM, editors. Case studies in needs assessment. SAGE Publications; 2019 Sep 5.
Domlyn AM, Wandersman A. Community coalition readiness for implementing something new: using a Delphi methodology. Journal of community psychology. 2019 May;47(4):882 – 97.
Bertram RM, Blase KA, Fixsen DL. Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice. 2015 Jul;25(4):477 – 87.
Domlyn AM. Implementation stages in practice: a review of behavioral health innovation within hospitals. Health Services Management Research. 2022 May;35(2):92–109.
Brady SR. The Delphi Method. In: Jason L, Glenwick D, editors. Handbook of methodological approaches to community-based research: Qualitative, quantitative, and mixed methods. Oxford university press; 2016.
Reid N. The Delphi technique: its contribution to the evaluation of professional practice. Professional competence and quality assurance in the caring professions. 1988;230:262.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing. 2000 Oct;32(4):1008-15.
Fletcher AJ, Marchildon GP. Using the Delphi method for qualitative, participatory action research in health leadership. International Journal of Qualitative Methods. 2014 Feb;13(1):1–8.
Kezar A, Maxey D. The Delphi technique: An untapped approach of participatory research. International journal of social research methodology. 2016 Mar 3;19(2):143 – 60.
Trevelyan EG, Robinson N. Delphi methodology in health research: how to do it?. European Journal of Integrative Medicine. 2015 Aug 1;7(4):423-8.
Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and policy in mental health and mental health services research. 2015 Sep;42(5):533 – 44.
Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American journal of community psychology. 2008 Jun;41(3):171–81.
Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research. 2011 Jan;38(1):44–53.
Hasson F, Keeney S. Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change. 2011 Nov 1;78(9):1695 – 704.
Kornbluh M. Combatting challenges to establishing trustworthiness in qualitative research. Qualitative research in psychology. 2015 Oct 2;12(4):397–414.
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC medical research methodology. 2013 Dec;13(1):1–8.
Miles MB, Huberman AM, Saldaña J. Qualitative data analysis: A methods sourcebook. Sage publications; 2018 Dec 13.
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, Wales PW. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of clinical epidemiology. 2014 Apr 1;67(4):401-9.
Sullivan GM, Artino AR Jr. Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013 Dec;5(4):541-2.
Zimmermann HJ. Fuzzy set theory—and its applications. Springer Science & Business Media; 2011 Jun 27.
Smith JD, Polaha J. Using implementation science to guide the integration of evidence-based family interventions into primary care. Families, Systems, & Health. 2017 Jun;35(2):125.
Chokron Garneau H, Assefa MT, Jo B, Ford JH, Saldana L, McGovern MP. Sustainment of Integrated Care in Addiction Treatment Settings: Primary Outcomes From a Cluster-Randomized Controlled Trial. Psychiatric Services. 2022 Mar 1;73(3):280-6.
Mautone JA, Wolk CB, Cidav Z, Davis MF, Young JF. Strategic implementation planning for integrated behavioral health services in pediatric primary care. Implementation research and practice. 2021 Jan;2:2633489520987558.
Ramanuj PP, Talley R, Breslau J, Wang SS, Pincus HA. Integrating behavioral health and primary care services for people with serious mental illness: a qualitative systems analysis of integration in New York. Community mental health journal. 2018 Nov;54(8):1116-26.
Wolk CB, Last BS, Livesey C, Oquendo MA, Press MJ, Mandell DS, Ingram E, Futterer AC, Kinkler GP, Oslin DW. Addressing common challenges in the implementation of collaborative care for mental health: the Penn Integrated Care program. The Annals of Family Medicine. 2021 Mar 1;19(2):148 – 56.
Heath BW, Wise Romero P, Reynolds K. A standard framework for levels of integrated healthcare. Washington, DC: SAMHSA-HRSA Center for Integrated Health Solutions. 2013 Mar 6.
Metz A, Bartley L. Active implementation frameworks for program success. Zero to three. 2012 Mar;32(4):11 – 8.
Leutz W. Reflections on integrating medical and social care: five laws revisited. Journal of Integrated Care. 2005 Oct 1.
Collins C, Hewson DL, Munger R, Wade T. Evolving models of behavioral health integration in primary care. New York: Milbank Memorial Fund. 2010 May;504:1–88.
Scaccia JP, Cook B, Wandersman A. Building Organizational Readiness (Capacities x Motivation) for Implementation: A Research Synthesis of the Empirical Evidence. Epub ahead of print 12 April 2020. DOI: 10.31235/osf.io/84cjq.
Stover AM, Wang M, Shea CM, Richman E, Rees J, Cherrington AL, Cummings DM, Nicholson L, Peaden S, Craft M, Mackey M. The Key Driver Implementation Scale (KDIS) for practice facilitators: Psychometric testing in the “Southeastern collaboration to improve blood pressure control” trial. PloS one. 2022 Aug 24;17(8):e0272816.
Barraclough F, Smith-Merry J, Stein V, Pit S. Workforce development in integrated care: a scoping review. International journal of integrated care. 2021 Oct;21(4).
Palinkas LA, Campbell M, Saldana L. Agency leaders' assessments of feasibility and desirability of implementation of evidence-based practices in youth-serving organizations using the stages of implementation completion. Frontiers in Public Health. 2018 May 29;6:161.

Download PDF

Version 1

posted

You are reading this latest preprint version

Implementation stages for integrating behavioral health and primary care: A participatory Delphi study to refine a determinant framework

Status:

Version 1

Abstract

Contributions To The Literature

Background

Methods

Results

Final Results

Discussion

Generalizability beyond integrated care

Strengths and Limitations

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1