Aims
The overall aims of the proposed study are to examine technology-driven approaches to implementing SEEK and to understand facilitators and barriers regarding its implementation and short-term sustainment, while also examining the effectiveness of these strategies. Doing so will advance knowledge in implementation science related to primary care and the prevention of CM. The randomized Type III hybrid design (66) leverages a commitment by 5 major healthcare systems to implement SEEK, enabling a rigorous evaluation of implementation strategies to optimize the adoption and delivery of SEEK in primary care settings, and subsequent prevention of CM: the independent online (IND) (59–61) versus in-depth structured MOC training (62–64). We will also observe the impact of using the SEEKonline and the Traditional approach to guide fidelity of model delivery. Further, SEEK will be examined in pediatric and family medicine settings, increasing the generalizability of findings.
Aim 1: Evaluate the effectiveness of targeted implementation strategies on the implementation of SEEK in primary care settings. Practices will be randomly assigned to one of two training conditions (IND or MOC). H1. MOC training will lead to more positive PCP attitudes, comfort level and competence in addressing risk factors for CM. H2. Users of SEEKonline will deliver the intervention more often, achieving a higher rate of penetration, and will report higher levels of provider and parent satisfaction than the Traditional mode of delivery. H3. MOC training and SEEKonline will together optimize adoption and sustainment of SEEK.
Aim 2. Evaluate the impact of inner context variables (e.g., variation between pediatric and family medicine) on the SEEK implementation process and understand associated barriers and facilitators to successful service start-up and sustainment of SEEK delivery. Using a mixed methods approach, standardized measures of the implementation process (Stages of Implementation Completion; SIC) and associated cost (Cost of Implementing New Strategies; COINS), will be integrated with qualitative interview data focusing on barriers and facilitators during implementation from exploration to sustainment. Variations in adoption, model fidelity, and sustainment, and the economic ramifications of the SEEK training and implementation strategies will be examined.
Aim 3. Examine the effectiveness of the intervention strategies in preventing CM. CM will be measured via prevalence of ICD-10 codes related to CM obtained from EHRs for all children 0-5 attending the practices. H1. Incidence of CM will be reduced in practices after implementing SEEK. H2. Practices randomized to MOC training that successfully implement SEEK, will have lower incidences of CM than with the IND approach. We will also observe the influence of SEEKonline and the Traditional approach on CM rates. Additionally, implementation success, will be examined in relation to CM prevalence rates.
INSERT FIGURE 2 ABOUT HERE
Study Design (Figure 2).
The study uses a rigorous Hybrid Type III design to examine the effectiveness of technology driven training strategies to facilitate SEEK’s adoption and implementation in pediatric and family medicine settings, and helping prevent CM. The implementation approach is anchored in four stages of the EPIS framework: Exploration, adoption/Preparation, Implementation, and Sustainment (67).
Exploration and Sample. Leaders in five healthcare systems that strongly supported the grant application will be approached to formally approve participation, without committing individual practices. They include 59 practices with 306 PCPs. Different regions provide diversity in terms of urban, suburban and rural locations, and racial/ethnic diversity. Three systems have integrated behavioral health professionals. We plan to study 13 pediatric and 35 family medicine practices. In addition to five system leaders, we will recruit 5 groups of participants: 1) 48 practice leaders, 2) 275 PCPs, 3) 16 behavioral health professionals, 4) 96 ancillary staff, and 5) 288 parents (6 per practice). Smaller subsamples will be purposively selected for qualitative interviews according to practice type, training strategy and presence of integrated behavioral health.
Inclusionary criteria are: 1) practices not already implementing SEEK, 2) practices providing primary care to children, and 3) agreement to participate. A letter will be sent to physician leaders of practices inviting participation in the study. If interested, we will hold a 1-hour webinar with the practice leader, PCPs and behavioral health professionals and key office staff. We will provide written material detailing what their involvement will entail. Participation will be voluntary, and, while we prefer that all the PCPs in a practice participate, this will not be required. We anticipate that behavioral health professionals and office staff will necessarily be involved in practices opting to participate. Parents will be recruited via flyers in the waiting area. If interested, they will be asked to notify staff who will request permission to convey their contact information to SEEK project staff. We will contact them and explain the project by phone and in writing.
Adoption/Preparation. We will sign an MOU with participating practices and request a limited informed consent by practice leaders and PCPs who opt out to gather data influencing their decision. We will identify a physician “champion” and an office staff member to lead implementation in each practice. Practices will be randomized to one of the two training strategies (IND vs. MOC). Practices, however, will be able to select the facilitation strategy (SEEKonline or Traditional) as we are unable to require this of participants. The design accounts for heterogeneity in geography, size of healthcare systems, type of primary care (pediatric and family medicine) and presence of integrated behavioral health. As shown in Figure 2, professionals, office staff and parents are nested within practices which are nested within the 5 healthcare systems. To ease introduction of the SEEK model, we will address logistical issues such as for which checkups parents will complete the SEEK PQ-R and documentation in the EHR, and the SEEK Parent Handouts will be customized with information on local resources.
Implementation. Informed consent will be obtained online from all participants at the beginning of the baseline survey. Practice leaders, PCPs, behavioral health professionals and office staff will be asked to complete the surveys (see Table 3) prior to the PCP training. Subsamples will be selected for 30-minute phone qualitative interviews. At the end of training, PCPs will evaluate it. Interested practices will have SEEKonline connected to their EHR, ensuring interoperability. Procedures for addressing concerns of possible CM will be aligned with federal, local, and professional guidelines.
Sustainment. Participants will be assessed at 12-, 21- and 33-months post-baseline regarding their thinking, feeling, behavior and experiences (see Table 3) to help inform understanding of SEEK’s implementation and sustainment. Final surveys and phone interviews will assess participants’ thoughts and plans about continued use of SEEK. Following the grant period, practices that wish to continue utilizing the implementation strategies will be guided through the process of establishing real-world, not-grant funded contracting.
Final 12 months. De-identified aggregate data will be abstracted from practices’ EHRs on all children (0-5) for the periods prior to and during SEEK implementation: rate of eligible visits where screening occurred, types of problems identified, types of actions taken, receipt of services, as well as CM-related diagnoses using ICD-10 codes. This will be facilitated by 4 of the 5 systems using Epic as their EHR and having in-house IT. The data will be analyzed, the Final Report prepared, and presentations and papers will continue to be developed.
Measures (See Table 3)
Measurement will include survey-based assessments, targeted qualitative interviews, observational implementation assessment, and EHR data on service delivery of SEEK (e.g., rate of screening) and CM.
INSERT TABLE 3 ABOUT HERE
Organizational Assessment
Participants will be asked to complete the SEEK Adoption Survey and the following four measures (1) Practice Demographics Form, (2) the Evidence-Based Practice Attitude Scales measuring attitudes toward adopting EBPs (68), (3) the Implementation Leadership Scale assessing leader support for EBPs within an organization (69), and (4) the Implementation Climate Scale measuring how an organization views new interventions (70). A subsample of participants will undergo a semi-structured, 30-minute phone interview.
Implementation Process and Outcomes
The implementation process and outcomes will be measured primarily using the Stages of Implementation Completion (SIC), an observational assessment tool (10,71,72). The SIC has 8 stages, each including sub-activities, extending from Engagement with the developers to achievement of practitioner Competency. Sustainment is an ongoing process toward achievement of Stage 8. Multiple adaptations of the measure have led to the development of the empirically derived Universal SIC, with items found to be relevant and reliably utilized regardless of service sector or population. For the current study, completion of activities will be monitored by the practice manager and collected monthly by the research team, with data entered into the SIC data collection website. Three scores are calculated for each SIC stage. First, the time that a practice takes for a stage is calculated (Duration Score). Second, the proportion of activities completed within a stage is calculated (Proportion Score). Third, the SIC Stage Score marks the final stage that a site reaches. SIC scores are calculated within each of the three Implementation Phases: Pre-implementation (stages 1-3), Implementation (stages 4-7), and Sustainment (stage 8).
The SEEK Adoption Survey will capture key influences on the decision whether to adopt SEEK, including perceived barriers and strengths. The SEEK PCP Training Evaluation Form will evaluate PCPs’ perceptions of the training approaches and solicit input as to how the training might be improved. The SEEK PCP Questionnaire, used in both SEEK RCTs (39,40), assesses PCPs’ thinking and practice with regard to addressing the targeted CM risk factors. We will use the SEEK PCP Survey to assess PCPs’ experience implementing SEEK including their perceptions of its relevance, ease of delivery, helpfulness and training. The SEEK Office Staff Survey will assess staff experiences with SEEK, including their understanding of their roles and perceived competence in implementing the model. The Rate of Screening for the targeted problems will be abstracted from EHR data, as will be the Receipt of Services by parents with positive screens. The SEEK Parent View of Child’s PCP, adapted for pediatric practice from the Patient-Doctor Interaction Scale will be used to assess change in parent’s views of PCPs associated with stage of implementing SEEK (73).
Child Maltreatment
EHR CM-related Diagnoses ICD-10 codes accessible through EHRs (74). De-identified aggregate data will be gathered toward study end for all children 0-5 attending the practices during the study – for up to 2 years prior to and during the study.
Qualitative Assessment: Phone Interviews
To assess potential barriers and facilitators of the SEEK intervention, we will conduct 30-minute semi-structured phone interviews with purposively selected (based on role, practice type, training strategy and presence of behavioral health) subsamples of 5 groups: system and practice leaders, PCPs, behavioral health professionals, office staff, and parents, at 3 time points. The first 3 groups will be interviewed around the start of the intervention (with a focus on adoption) and 11 and 21 months later (with a focus on implementation and maintenance). The last 2 groups will be interviewed at 11-, 20- and 32-months following start of the intervention, after they have experienced SEEK. Interviewers will use a semi-structured guide using the EPIS framework to align questions with stage of implementation (75). Participants will be asked about their experiences with SEEK, assessment of training and implementation support, challenges in delivering the intervention and recommendations for addressing the challenges. Interviews will be digitally recorded and professionally transcribed for analysis.
Measuring Costs
The cost analysis will provide estimates of the overall practice-level costs associated with implementing SEEK. Cost measurement will be organized using the Cost of Implementing New Strategies (COINS) framework which provides a structure for measuring and categorizing costs (76). Implementation costs include all resources used to deliver SEEK. COINS maps onto the SIC, by tracking the costs and resources needed to complete each implementation activity. In SEEK II, PCPs in the intervention arm did not require more time per child than did controls. It thus seems reasonable to apply this to the current study, rather than conduct another time study.
Data Analysis Plan
General Considerations. Standard statistical methods for calculating point estimates, confidence intervals, and p-values require the assumption of independence. However, due to participants being clustered within healthcare systems and within practices, this assumption cannot be made. To account for the lack of independence among multiple measures within the same system or practice, we will use mixed effects (i.e., hierarchical) models. In these models, we will include random effects for system and for practice. In analyses involving multiple measures from the same person, we will also include a random effect for person. Below we highlight the main analyses for each Aim.
Aim 1.Compare alternative approaches to implementation of SEEK with respect to clinical and implementation outcomes. To address aim 1a, we will compare practices randomized to MOC to practices randomized to IND with respect to clinical and implementation outcomes. These include PCPs’ perceptions of the training, scales from the SEEK PCPQ (e.g., Competence in addressing problems) and the PCP Survey (e.g., Ease of Delivery). We will also compare the groups’ rates of screening and parents’ receipt of services and satisfaction with PCPs. Statistical inference will be based on mixed effects models fitted using restricted maximum likelihood. To avoid possible biases due to selective attrition, the primary analysis will follow the “intention to treat” principle including all those randomized. Secondary analyses will be based on groups defined by training received. To address Aim 1b we will examine practices who choose SEEKonline and those choosing the Traditional approach using the same statistical methods as for Aim 1a. Outcomes of interest will include staff satisfaction, costs, rates of screening, receipt of services and parental satisfaction.
Aim 2. Examine variations in SEEK’s implementation process and impacts and understand associated barriers and facilitators in pediatric and family medicine practices. We will determine the proportion of practices that agree to adopt SEEK. We will then assess the relationship between practice characteristics (e.g., demographics, EBP attitudes) and willingness to adopt SEEK. The most important independent predictors will be determined using multivariable logistic regression models. Among those that do adopt SEEK, we will assess their degree of completeness, speed and quality of implementation using the SIC. Completeness will be summarized using the final stage attained (0-8). Speed will be summarized by the duration in each stage, and quality by the proportion of activities performed at each stage, and overall. The distribution of time to achieve each phase will be estimated using the Kaplan-Meier approach. Barriers and facilitators to implementing SEEK will be probed quantitatively and qualitatively. Among practices adopting SEEK, we will examine the association between practice characteristics and measures of completeness, speed and quality of implementation from the SIC using multivariable mixed effects models.
Aim 3. Examine the effectiveness of SEEK in reducing CM. As in our previous studies (37,38), we will identify CM-related diagnoses, now via EHRs. We will review the EHR for each child (0-5) in each practice for up to two years prior to implementing SEEK and during implementation. The presence of CM-related ICD-10 codes will be recorded. The proportion of children with CM diagnoses before and during SEEK will be compared at each practice. Formal inference regarding the best estimate and statistical significance of pre-post differences will be based on a binary regression model with a random effect for site, similar to Aim 1.
Cost Analysis. Total practice cost of implementation completion will be calculated for both training arms, not for implementing the entire SEEK model. Cost effectiveness ratios will be calculated as the cost per average SIC component completed, and per average Competence scale and Practice Behavior scale scores. Standard time discounting methods will be applied to cost estimates (77). We will use sensitivity analyses to derive upper and lower estimates of resource use and implementation costs (77). Standard errors for use in mean comparisons will be estimated, using bootstrapping methods (77).
Qualitative/Mixed MethodsAnalysis. We will keep an audit trail of data collected and memos, team meetings indicating time, place, source of data, and persons collecting or analyzing information. We will analyze interview transcripts using a thematic content analysis methodology (78). First, transcripts will be reviewed by investigators to develop a broad understanding of content related to the project’s aims and to identify topics for discussion and observation. Second, segments of text ranging from a phrase to several paragraphs will be assigned codes based on a priori (i.e., from the interview guide) or emergent themes (or, open coding) (79). Codes will be assigned to describe connections between categories and between categories and subcategories (i.e., axial coding) (79). Codes will also be assigned to reflect participants’ social and demographic characteristics. Lists of codes developed by each investigator will be matched and integrated into a single codebook. Third, each text will be independently coded by at least two investigators. Disagreements in assignment of codes will be resolved through discussion between investigators and by refining definitions of codes. With the final coding structure, two investigators will separately review transcripts to determine level of agreement. A level of agreement ranging from 66-97% depending on level of coding (general, intermediate, specific), indicates good reliability in qualitative research (80). Fourth, based on these codes, the computer program QSR NVivo will generate a series of categories arranged in a treelike structure connecting text segments grouped into separate categories of codes or “nodes” (81). These nodes and trees will be used to further the process of axial or pattern coding to examine the association between different a priori and emergent categories. Fifth, by constantly comparing these categories with each other, the different categories will be further condensed into broad themes using a format that places SEEK’s effectiveness and implementation within the framework of the system characteristics (82). Finally, the themes will be compared with the results of the analysis of quantitative data relating to PCP experiences with SEEK to identify points of convergence and divergence (triangulation) and to explain potentially unanticipated findings (expansion).