Our formative evaluation involved 22 participants, including 11 potential interventionists (8 IBH therapists, 3 community wellness advocates) and 11 stakeholders (6 PCPs, 5 clinic leaders and administrators representing psychology, psychiatry, social work, and operations). No participants refused to participate or dropped out. Sociodemographic information was not collected to preserve employee privacy. Interviews were completed by all participants, though only IBH therapists (our primary interventionists) were asked to comment on appropriateness of the proposed intervention. Few stakeholders had familiarity with the intervention, therefore were asked to comment more generally on its key features (e.g, five-session, cognitive behaioral therapy, brief PTSD intervention).
Surveys
Table 3 provides descriptives of survey findings. Overall, participants noted strong organizational support (CFIR inner setting) for implementation of evidence-based practices (ICS Total M = 2.06, SD = 0.42), with ratings similar or higher than ratings of implementation climate observed in other mental health (38) and substance use settings (48) Participants reported low ratings on the ICS subscales staff selection (M = 1.74, SD = 0.77) and rewards (M = 0.62, SD = 0.64) subscales, suggesting that although participants favorably rated their organization’s attitudinal support of evidence-based practices, the organization does not explicitly seek to hire staff who are trained in or value evidence-based practices (CFIR characteristics of individuals involved), nor does the organization provide financial incentives for adoption of these approaches (CFIR inner setting). These findings were slightly below average for selection and similar for rewards compared to other studies utilizing ICS (38,48,49). Among IBH participants (n = 13; 11 potential interventionists and 2 stakeholders), there was strong attitudinal support for adoption of evidence-based behavioral health treatments (EBPAS M = 2.8, SD = 0.72), with ratings comparable to those observed among community mental health service providers (39,49). Overall, participants reported slightly positive ratings of behavioral health integration (LIM M = 3.51, SD = 0.37), which were comparable to other safety net settings (50) but indicate the need for improvement in behavioral health integration. Importantly, survey scores did not significantly vary between primary care and IBH clinician participants (Table 4), emphasizing that priorities for program implementation for PTSD were similar across stakeholder type. Indeed, a joint display comparison of high and low survey scores shows shared sentiments across both primary care and IBH individuals from the ICS and LIM (Table 5).
Table 3
Descriptive data from surveys
Measure
|
Response Format
|
Item Content
|
Subscale
|
M (SD)
|
All Participants (N = 22)
|
Levels of Integration Measure (LIM) (5 Items, 1–5 Scale)
|
Evaluative
(Strongly Disagree to Strongly Agree)
|
Assessment of integration between PCPs and IBH clinicians (answered by those in primary care and IBH)
|
Integrated clinical practice
Systems integration
Training
Relationships
Shared decision making
Beliefs and commitment
Leadership
Overall
|
3.47 (0.62)
3.23 (0.38)
3.20 (0.71)
3.76 (0.60)
3.26 (0.63)
4.23 (0.45)
3.70 (0.89)
3.51 (0.37)
|
Implementation Climate Scale (ICS)
(5 Items, 0–4 Scale)
|
Evaluative (Not At All to A Very Great Extent)
|
Assessment of agreement with statements reflecting three organizational climate dimensions of innovation use (answered by those in primary care and IBH)
|
Focus on EBT
Educational support for EBT
Recognition for EBT
Rewards for EBT
Selection for EBT
Selection for openness
Overall
|
2.78 (0.62)
2.59 (0.94)
2.22 (0.85)
0.85 (0.71)
1.78 (1.05)
2.41 (0.81)
2.09 (0.81)
|
IBH-Level Participants (N = 13; 11 potential interventionists, 2 behavioral health leadership)
|
Evidence-Based Practice Attitude Scale (EBPAS)
(5 Items, 0–4 Scale)
|
Evaluative (Not At All to A Very Great Extent)
|
Assessment of agreement with attitudes towards EBTs (answered by those in IBH)
|
Requirements
Appeal
Openness
Divergence
Overall
|
2.64 (1.34)
3.04 (1.08)
2.67 (0.72)
2.81 (0.74)
2.80 (0.72)
|
Note. PCP, primary care physician; IBH, integrated behavioral health; EBT, evidence-based treatment. Joint display of survey results with qualitatively derived codes for evaluation of integration and organizational climate amongst primary care and IBH.
|
Table 4
Comparison of survey scores by primary care and IBH participants
Survey
|
Clinic Type
|
M (SD)
|
Difference between clinic type (P-Value)
|
|
ICS
|
Primary Care (n = 9)
|
37.67 (9.19)
|
0.82
(p > 0.05, Not Significant)
|
|
|
IBH (n = 13)
|
36.85 (6.67)
|
|
|
LIM
|
Primary Care (n = 9)
|
121.44 (9.34)
|
0.79
(p > 0.05, Not Significant)
|
|
|
IBH (n = 13)
|
122.85 (14.86)
|
|
Note. IBH, integrated behavioral health; ICS, implementation climate scale (38); LIM, levels of integration measure (39). ICS total score out of 72, and LIM total score out of 175. |
Student t-test with Welch correction comparing survey response between primary care and IBH provider. |
Table 5
Quotes from participants corresponding with high and low survey scores
Survey Rating
|
Clinic Type
|
Examples
|
Treatment Plans for PTSD to Meet Patient Needs
Knowledge and training for identifying trauma and PTSD; treatments systematically meet the needs of patients
|
|
High ICS Scores
|
IBH
|
“When we [IBH] go through the psychiatric review of symptoms, I always ask in the intake, ‘is there a traumatic event that you witnessed that is still upsetting you today?’”
|
|
Low ICS Scores
|
Primary Care
Primary Care
|
“I think one of the challenges is that I don’t find that my colleagues in primary care are particularly trauma-literate, so I really believe that we have to do some more education for other staff in our clinic.”
“I have to say, my familiarity with PTSD is not as strong.”
|
Support for Evidence Based Practices
Training, workshops, or materials about applying EBT for treatment; positive attitudes and champions for evidence based practice within clinics; integration between primary care and IBH
|
|
High LIM Scores
|
Primary Care
|
“I think the community health workers would be very enthusiastic about it [manualized PTSD therapy].”
|
|
Low LIM Scores
|
IBH
Primary Care
|
“I think that we [IBH] are very siloed out in family medicine, kind of out of people’s radar.”
“Much of the energy has been focused on meeting what we [primary care] have to do…to deliver [PTSD] program[s] in the eyes of [state insurance programs]. There’s a tremendous amount of work left to be done, like optimizing care and delivery.”
|
Note. PTSD, posttraumatic stress disorder; IBH, integrated behavioral health; ICS, implementation climate scale (38); LIM, levels of integration measure (39); EBT, evidence-based treatment. Joint display that relates categorical survey results from ICS and LIM to quotes from primary care and IBH stakeholders. |
Interview Findings
Although our interview guide focused on barriers to implementation of a brief cognitive behavioral therapy for PTSD (STAIR-PC), responses were often pertinent to the overall provision of brief mental health interventions in the integrated primary care setting, and not specific to PTSD treatment. See Table 6 for more detail on CFIR constructs that we identified as influencing implementation, and initial recommendations generated through interviewee feedback and community advisory board engagement.
Table 6
Factors that influence implementation of an EBT for PTSD in safety net primary care
CFIR Constructs
|
Definition
|
Examples
|
Intervention Characteristics
|
|
|
Relative Advantage
|
Stakeholders’ perception of advantages of implementing the five-session, manualized intervention versus an alternative solution.
|
Positive attitudes about the intervention compared to alternative EBTs (brevity, flexibility, transdiagnostic application, effectiveness) and compared to current standard of care (primarily psychoeducation, relaxation training only)
Although some patients may need higher intensity therapies, this is an effective lower intensity treatment which may have engagement advantages.
|
|
Adaptability
|
The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs.
|
Interventions needs to fit current practice, including session length and scheduling frequency.
|
Outer Setting
|
|
|
|
Patient Needs and Resources
|
The extent to which patient needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the organization.
|
May experience challenges with prioritizing PTSD-focused treatment in the context of social determinants of health and related barriers to engagement (transportation, caregiving responsibilities, socioeconomic status)
|
Inner Setting
|
|
|
|
Networks and Communications
|
The nature and quality of webs of social networks and of formal and informal communications within an organization.
|
Challenges with internal and external referral processes for behavioral health treatment and the inability of PCPs to access therapy notes has led to issues with care coordination.
Challenges unique to a teaching hospital, where staff rotate and turnover frequently, has resulted in problems with communication. Need for ongoing and repeated training and communication about treatment options and clinic processes.
|
|
Relative Priority
|
Individuals’ shared perception of the importance of the implementation within the organization.
|
ACO has led to organizing around depression care management; PTSD is not currently a quality metric. This is both a barrier and facilitator, since some of the improvements are good for all mental health services (e.g., routine depression screening).
|
|
Leadership Engagement
|
Commitment, involvement, and accountability of leaders and managers participating in the implementation.
|
Need to gain support from operations managers and population health leaders in the practice for implementation success, especially in regard to protecting time and modifying workflows to accommodate EBTs.
|
|
Available Resources
|
The level of resources dedicated for implementation and on-going operations, including money, training, education, physical space, and time.
|
By nature, IBH prioritizes access (brief screening, treatment referral); however, accommodating routine appointments typical for EBTs can be a challenge.
Due to time and resource burdens, screening is not feasible in most PCP or IBH therapist visits. Need to consider the role of medical assistants in initial screening for PTSD.
Difficulties securing protected time for training and consultation.
Need to embed researcher on clinical support time for training purposes and to respond to implementation challenges in real-time.
|
Characteristics of Individuals - Providers
|
|
|
Knowledge and beliefs about the intervention
|
Individuals’ attitudes toward and value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention.
|
PCP knowledge gap of best practices for assessment, treatment options, and referral process for PTSD (e.g., rely on heuristics of stereotypes to determine who needs assessment; asking too much detail about trauma events in screening process; lack of knowledge of referral pathways).
PCPs unsure of their ability to assess for PTSD. Express concerns about distressing patients and not knowing how to respond or support.
PCPs expressed positive attitudes about co-learning with IBH therapists and how this project may foster improvements in collaborative care.
Most IBH therapists expressed that trauma-focused CBT is a good fit for their patients, especially if it can be delivered in a brief format.
Some IBH therapists expressed attitudes that brief treatments are not appropriate for PTSD (requires long-term therapy).
|
Characteristics of Individuals - Patients
|
|
|
Other personal attributes
|
A broad construct to include other personal traits such as tolerance of ambiguity, intellectual ability, motivation, values, competence, capacity, and learning style.
|
Emergent Theme: Mental health stigma, language, literacy, care setting preferences, patient treatment priorities, especially in context of social determinants, were mentioned as important considerations.
|
Note. CFIR, consolidated framework for implementation research (32); EBT, evidence-based treatment; PTSD, posttraumatic stress disorder; ACO, accountable care organization; IBH, integrated behavioral health; PCP, primary care physician; CBT, cognitive behavioral therapy. |
CFIR: Intervention Characteristics
Relative Advantage. Respondents affirmed that a brief five-session manualized intervention was appropriate for use in primary care and was considered a more advantageous intervention than current practices for several reasons. As an IBH provider shared that, relative to other EBTs for PTSD which are longer and more intensive, the structure of a brief manualized treatment would help orient patients to the short-term approach to PTSD therapy while also providing effective care: “if we’re [IBH] already planning for five sessions, then we [IBH] can market it [manualized PTSD intervention] that way for patients when they come in for the intake... plus, manualized treatments are super convenient.”
They expressed how low intensity and brief treatment in IBH may be more suitable for patients seen in primary care, and especially for patients with barriers to engagement in high intensity specialty care (e.g., stigma, mental health literacy, affective avoidance, or suppression). They hypothesized that patients may be more open to engage in stepped-up care if needed after STAIR-PC, as one PCP stated: “[for] people who will never ‘land’ in [specialty care], if we give this [manualized PTSD treatment] and they have a good experience, they may be more willing to do the next level.”
Adaptability. Respondents shared how the proposed intervention could fit the local practice with some modifications. As one PCP noted, “[physicians] want this [type of therapy]. [When we] have a very short [and] effective way to treat PTSD — where we teach people these skills they can practice at home to reduce their symptoms — [PCPs] can [offer this treatment option] and keep an eye on [treatment progress].” One IBH provider described how STAIR-PC was an appropriate fit based on patient needs and the IBH setting: “I think [a manualized PTSD treatment] is in line with the other interventions that we are trying to offer in IBH. So I think [the treatment fits the setting] completely. Both within an IBH model and for our patient population, where I think, you know, our focus tends to be on how do we help improve their functioning right now in their lives? So I think it fits within the scope of the work that we're doing, and I don't see any reason why it wouldn't be a good fit for this patient population.”
CFIR: Outer Setting
Patient Needs and Resources. Respondents noted that socioeconomic determinants of health were common barriers to engagement in and prioritization of PTSD treatment. As one IBH provider explained, “[engagement] could be challenging given the complexity of working with [patients experiencing] …homelessness or financial instability.” Additionally, respondents described transportation and caregiving barriers that contribute to late or missed appointments, resulting in less provider contact and high dropout. As one PCP emphasized, “the biggest thing for a lot of patients is competing priorities, living in a world where finding a job and being safe is the first priority and convincing people that it [mental health] is worth their time [comes second].”
CFIR: Inner Setting
Networks and Communications. Respondents emphasized the importance of care coordination, including the need for clear internal and external referral processes between clinicians within the primary care setting.
Referrals to IBH. The process for referring to IBH utilized warm handoffs. Several mentioned that, while ideal, warm handoffs can be logistically difficult: “I think there’s a kind of subtle disincentive to use [a warm handoff] in that no matter how fast it [a warm handoff] is, you’re grabbing a room and you’re taking time, [an] extra 10 or 15 minutes to have that warm handoff take place.” Despite these challenges, the majority of providers supported the use of warm handoffs and recognized their role in improving patient engagement. As another PCP explained, “the opportunity to do long warm handoff[s] will definitely increase that [patient] buy in.”
Referrals to Specialty Care. By contrast, the referral process to specialty care is less specified and rarely utilizes warm handoffs. Respondents reported that the practice of relying on patients to spontaneously disclose trauma history or PTSD symptoms during routine primary care visits may miss patients with mild-to-moderate PTSD symptom severity or those who are reluctant to disclose. High severity referrals going to IBH, and access issues when attempting to refer out to specialty care led some respondents to believe that patients seen in IBH may be higher in symptom severity than typically seen in IBH models of care. IBH respondents also voiced reservations about referring out to specialty care, noting concerns about patient dropout or uncertainty about current therapies offered in specialty care. One IBH provider described the siloed nature of IBH and specialty care: “I’ve only ever been to [the specialty care clinic] once or twice, and I don’t know that many people who work there, so…I have very little idea what actually goes on over there.”
Relative Priority. At the time of this evaluation, the hospital was shifting to an Accountable Care Organization (ACO), which links reimbursements to quality metrics and reductions in the cost of care. Some respondents felt that the central focus on ACO quality metrics may pose challenges to adding a screener for PTSD. Under the ACO, depression care screening is prioritized, and hospitals are evaluated, scored, and provided funding based on the frequency and consistency of PHQ-2 and − 9 administration. Other respondents felt positively that the ACO shift may be a stepping stone to improving PTSD outcomes. One PCP noted how the redesigned system prompted by the ACO may allow greater access to behavioral health services, stating, “one of the real opportunities [of] the ACO is in restructuring the way that we think about [the] clinic more broadly and more aggressively than we thus far have.”
Leadership Engagement. Respondents also noted that buy-in from operations managers and population health leaders was critical to successful implementation, including securing protected time and resources to support adoption. One PCP noted the importance of regular communication with leadership: “there's a lot going on, and…. [as part of the leadership team] … I have a good handle of all the programs ...but, I think I'm probably the minority. The PCPs who are there once a week may not know of all the resources that are available to our patients.”
Available Resources. Workflow and productivity demand, documentation and case management tasks, and administrative meetings create extensive time pressures on all providers that limit availability for trainings needed for successful implementation. As such, respondents emphasized the need for additional protected time for training and suggested, in the meantime, that researchers capitalize on existing protected times for training.
Respondents also identified the availability of appointments for ongoing therapy as a major barrier to PTSD intervention implementation. Further, rescheduling missed appointments in a timely fashion is challenging due high patient volume, lowering accessibility and efficiency. Respondents emphasized the need to address these workflow challenges to accommodate EBT scheduling demands (at least two visits per month).
Respondents also noted the importance of embedding the researcher (study principal investigator) in the practice, with one IBH therapist stating, “I would find [it] really helpful [to have the principal investigator embedded in the clinical support team] because …if there’s anything wrong or different later [that impacts the roll-out], …that could help keep the momentum going.”
CFIR: Characteristics of Individuals Involved - Providers
Knowledge and Beliefs about the Intervention. Respondents described the need for educational trainings tailored to PCPs and IBH clinicians to improve understanding of trauma, PTSD diagnosis, and effective treatments. As one IBH provider expressed, “providers… just don’t know what to do with trauma.”
PTSD Assessment. Respondents explained how some providers applied variable heuristics and clinical judgement to decide which patients to screen for trauma and PTSD. PCPs described gathering a thorough trauma history with full details of index events, which is not the standard (or advised) process for screening. Others described screening for PTSD symptoms only when PCPs knew or suspected trauma exposure based on voluntary disclosure or patient characteristics (race and immigration status). As one PCP shared: “the only people I know [who] have PTSD are the ones who volunteer it [trauma history or diagnosis].” Respondents also described how some PCPs assign a PTSD diagnosis based on trauma exposure alone, without directly assessing for symptoms. Still, other PCPs may avoid asking patients about trauma due to concerns around distressing patients or insufficient time to respond to anticipated distress.
Current Treatment Options. PCPs also described lack of knowledge on the types of treatments available via each referral pathway, partly due to lack of access to therapy notes in the electronic medical record. PCPs shared their general enthusiasm for co-learning with their IBH colleagues and how this project might help them to stay abreast of PTSD treatment options and better meet the needs of their patients.
CFIR: Characteristics of Individuals Involved - Patients
Emergent Themes. Several themes regarding cultural considerations emerged as a subconstruct within this CFIR domain, reflective of the need for increased considerations of race and racism, stigma, shame related to mental health care, and language and literacy barriers among patients with PTSD. Respondents described how patients may be hesitant to disclose trauma and seek treatment for PTSD due to mental health stigma and stigma of referral to specialty care. As one community wellness advocate shared, “the word [name of specialty care clinic] is very stigmatized.” Respondents felt that a stepped care approach to PTSD treatment, with triaged levels, may help patients engage at their preferred setting and intensity of treatment.
Providers additionally described the role of race and racism in the patient-provider relationship. As one community wellness advocate stated: “a lot of patients have mentioned to me, especially if [they’re] a person of color, [that] they [patients] want to be able to relate [to] and see someone of color.” This perspective highlights how systems in which providers do not reflect diversity of patients may face additional challenges earning trust and engaging patients.
Contextualization of Findings
The community advisory board helped to contextualize interview findings, including providing more detail on current shortcomings of behavioral health referral processes. They made suggestions on how to improve referral pathways, including defining the scope of practice in IBH versus specialty care and provider education on referral procedures. The community advisory board also clarified that the hospital would be moving to open notes, allowing primary care physicians to better understand the focus of therapy and potentially assist with collaboration. Staff turnover is high in this setting, both due to the highly stressful clinical environment as well as the nature of teaching hospitals — therefore, they recommended that intervention trainings should be delivered in an ongoing fashion. The community advisory board explained that prioritization of PTSD screener was not feasible at this time, but agreed that it is a worthy long-term goal for the practice. They also made many suggestions to support therapist engagement, including blocking schedules for training and consultation and modifying clinical templates to allow for more regular visits.