Evaluating the national implementation of Bright IDEAS in clinical settings: applying the RE-AIM framework to characterize the clinical perspective

Background Bright IDEAS: Problem-Solving Skills Training (Bright IDEAS), a psychosocial intervention, has shown to improve problem-solving skills and reduce emotional distress in caregivers of children recently diagnosed with cancer. Though efforts have been made to scale up cancer control interventions, there are limited data assessing the adoption of Evidence-Based Cancer Control Programs (EBCCP) into clinical practice. This study describes the barriers and facilitators to implementation experienced by practitioners trained to use Bright IDEAS in their clinical settings. Methods A total of 209 Pediatric psychosocial oncology practitioners were trained through 10 workshops. Adaptations were made to the training agenda and curriculum based on practitioner feedback following implementation in local institutions. Practitioners were interviewed to understand their training experience and gain their perspective on the effectiveness of delivering Bright IDEAS in the clinical setting. The RE-AIM framework was used to guide the evaluation process we employed to assess the effectiveness of this national effort to disseminate Bright IDEAS. Results Interviews were conducted with 47 practitioners. Practitioners in the pre-adaptation group reported the identication of a particular prole of a client as a facilitator to intervention use. Additionally, perceived rigidity of the protocol, lack of consistency in intervention use with clients, feelings of being overwhelmed by the client as rationale for lack of interest, and lack of multiple trained practitioners at institutions were identied as barriers to intervention use. Practitioners in the post-adaptation group reported similar identication of a prole for an appropriate client for the intervention, more usage of Bright IDEAS materials, implementation across multiple clinical settings, and expressed continued commitment to intervention use as well as discussion of internal training for colleagues. Conclusions This study provides insight into how psychosocial practitioners adopt, implement, and maintain Bright IDEAS in the clinical setting and identied important considerations and needs for both practitioners and larger health care systems. It also provides some cautions to those who wish to promulgate evidence-based interventions. The EBCCP is designed to provide clinical program planners and public health practitioners with easy and immediate access to program materials. Despite efforts to scale up cancer control interventions, there are limited data assessing the adoption of survivorship/supportive care EBCCPs into clinical practice despite public availability through the database. This study examines the national implementation of a EBCCPs designated psychosocial intervention, Bright IDEAS: Problem-Solving Skills Training (Bright IDEAS), shown to improve problem-solving skills and reduce emotional distress in caregivers of children recently diagnosed with cancer. 5–8 Bright IDEAS received EBCC designation in 2013 and NCI tracking statistics indicate the Bright IDEAS webpage received approximately 470 views annually (averaging 2.7 minutes per page); only 40 CDs of the intervention materials were requested over a 7-year period. These ndings suggest that, despite NCI endorsement and the availability of Bright IDEAS through the EBCCs website for over 10 years and considerable evidence supporting its ecacy, the leap from research protocol to standard clinical care was likely not occurring. This study presents longitudinal follow-up ndings from qualitative interviews conducted among practitioners who participated in the Bright IDEAS training program, all of whom had committed to deliver the intervention at their respective institutions after receiving training to use Bright IDEAS. RE-AIM constructs were used to explore implementation barriers and facilitators. 9 The goal was to identify factors for improving the pragmatic application in real-world clinical settings to ensure future successful implementation.


Contributions To The Literature
We disseminated a psychosocial intervention nationally. This paper describes the evaluation of local implementation and results from practitioner interviews expressing best practices and necessary program adaptations for success.
This study helps address a gap in the literature by identifying key learnings from evaluating the application of a psychosocial intervention into routine clinical care.
Our ndings call for increased evaluation of intervention implementation of psychosocial interventions and highlights the need to identify program adaptations early and often to ensure success in real-world application.
Background Page 3/17 There is increasing urgency to address the gap between the generation of new knowledge and empirical evidence focusing on its application to routine clinical care. 1,2 This urgency is fueled, in part, by marginal awareness and variable adoption of effective interventions, commonly referred to as evidence-based practices (EBP) that research produces. Oftentimes, EBPs are tested under conditions that are different from the clinical practice settings for which they are designed. As a result, clinician-preferred interventions, many of which either lack empirical evidence or are delivered in a substandard manner, are preferred to EBPs. 2,3 In response to the limited uptake of EBPs in the clinical setting, the National Cancer Institute (NCI) created a searchable database of evidence-based cancer control programs known as Evidence-Based Cancer Control Programs (EBCCP; formerly, Research-Tested Intervention Programs (RTIPs)) (Evidence-Based Cancer Control Programs (EBCCP) website. National Cancer Institute. https://ebccp.cancercontrol.cancer.gov. Last Updated: 02/12/2020. Accessed on: 04/07/2021). The EBCCP is designed to provide clinical program planners and public health practitioners with easy and immediate access to program materials. Despite efforts to scale up cancer control interventions, there are limited data assessing the adoption of survivorship/supportive care EBCCPs into clinical practice despite public availability through the database. 4 This study examines the national implementation of a EBCCPs designated psychosocial intervention, Bright IDEAS: Problem-Solving Skills Training (Bright IDEAS), shown to improve problem-solving skills and reduce emotional distress in caregivers of children recently diagnosed with cancer. [5][6][7][8] Bright IDEAS received EBCC designation in 2013 and NCI tracking statistics indicate the Bright IDEAS webpage received approximately 470 views annually (averaging 2.7 minutes per page); only 40 CDs of the intervention materials were requested over a 7-year period. These ndings suggest that, despite NCI endorsement and the availability of Bright IDEAS through the EBCCs website for over 10 years and considerable evidence supporting its e cacy, the leap from research protocol to standard clinical care was likely not occurring.
To increase awareness of Bright IDEAS and facilitate adoption, an educational grant (R25CA65520) was obtained to train 200 pediatric psychosocial oncology practitioners from across the country and coach them in the use of the intervention. The main objective of the study presented here was to obtain information from practitioners trained to use Bright IDEAS; about the barriers and facilitators to implementation in their clinical settings. The speci c research questions for this study were: This study presents longitudinal follow-up ndings from qualitative interviews conducted among practitioners who participated in the Bright IDEAS training program, all of whom had committed to deliver the intervention at their respective institutions after receiving training to use Bright IDEAS. RE-AIM constructs were used to explore implementation barriers and facilitators. 9 The goal was to identify factors for improving the pragmatic application in real-world clinical settings to ensure future successful implementation.

Bright IDEAS Intervention
Bright IDEAS is a cognitive-behavioral coping skills training approach that allows caregivers to identify problems or challenges they are encountering that they wish to resolve (Fig. 1). No constraints are placed on the type of problem or challenge and, interestingly, the majority of selected problems are not related to pediatric cancer. [5][6][7] Optimum engagement is gained by focusing on problems the caregiver identi es as particularly relevant to him or her and the family. This approach also enables the caregiver and interventionist to review the effectiveness of selected solutions over time and changing circumstances. Notably, improved problem-solving skills mediates less distress. 6,8 Bright IDEAS has established clinical effectiveness among mothers of children with recently diagnosed cancer and has demonstrated dramatic and sustained improvement in caregiver mental well-being by decreasing negative affectivity as a result of improved problem-solving skills. 5,8 Clinical trials have shown that Bright IDEAS is acceptable to caregivers and, when used in six to eight 30-to 60-minute face-to-face sessions, enhances problem-solving skills and alleviates distress in mothers of children recently diagnosed with cancer. 5 In particular, Bright IDEAS has proven to be signi cantly more effective over time than the most common form of psychosocial support, non-directive supportive counseling. 10 Training The 1½-day training workshops were held in conjunction with association national meetings and endorsed through co-advertising. The workshops included summary information about the three large multi-site randomized controlled trials conducted to date demonstrating the e cacy of Bright IDEAS; role plays to observe and practice administration of the intervention; and in-depth discussions about implementation at an attendee's speci c home institution. Workshop participants received up to $1,000 to reimburse expenses associated with travel and lodging.
The original training agenda was modeled from the research training protocol and is detailed in Appendix A. After completing training, practitioners were asked about their perspective on their ability to adopt, implement, and maintain Bright IDEAS at their institution. The aim of the initial qualitative interviews was to identify factors for improving training delivery to improve implementation of Bright IDEAS in real-world clinical settings. After feedback from practitioners following initial implementation, adaptations were made to the training agenda which are presented in Appendix B. Table 1 summarizes the modi cations and their rationale. To facilitate implementation and promote trial, one-hour consultation conference calls were also held at 1, 2, 3, and 5 months after the workshop. Each call was attended by 3-5 workshop participants and led by 2 workshop leaders. All attendees were expected to participate and were asked to complete the intervention with at least three patients, caregivers, or patient-caregiver dyads over the 5-month course of the conference calls.
The practitioner's experiences were discussed on the calls in a supportive peer mentorship atmosphere.

Study Population
National Training Workshops We purposively sampled psychosocial practitioners who completed the Bright IDEAS training and delivered the intervention. A total of 209 practitioners representing 47 unique institutions completed the training. The interviews were scheduled to provide data from 3, 6, and 12 months following training in two waves to re ect the effect of modifying the workshop presentations in response to attendee suggestions. The project was approved by the Colorado Combined Institutional Review Board (COMIRB).

Practitioner Interviews
Study participant recruitment occurred between January 2017 and March 2020: Pre-adaptation: Wave 1-more than 12 months since training (N = 11), Wave 2-between 6 and 12 months (N = 9), and Wave 3-less than 6 months (N = 10). Post-adaptation: Wave 1-more than 12 months since training (N = 6), Wave 2-between 6 and 12 months (N = 4), and Wave 3-less than 6 months (N = 4). Practitioners were contacted via email and invited to participate in a 30-minute telephone interview. A random sampling of participants was contacted initially. Depending on response rate, the decision was made to continue or discontinue contact to secure additional interviews as needed. In total, 106 professionals were contacted, 68 responded to the study invitation (64% response), and 44 were scheduled (65% participation) for interviewing allowing up to three contact attempts.
Non-participation by three providers who originally agreed to participate was due to scheduling challenges. An honorarium of a $25 gift card was offered for participation.

Semi-structured interviews
Semi-structured interviews were conducted over the telephone in their o ce or other private setting by the rst author (DMM) who had no prior relationship with any of the respondents (range: 24-47 minutes per interview). All interviews were audio-recorded and transcribed verbatim.

Conceptual Framework For Evaluation
The RE-AIM framework, which is recognized by the NCI as a leading implementation framework in cancer control research, was used to guide the evaluation process we employed to assess the effectiveness of this national effort to disseminate Bright IDEAS. 9 The interview guide was developed to be consistent with dimensions of the RE-AIM framework and explored adoption, implementation, and maintenance of Bright IDEAS in the clinical setting. Table 3 gives example questions from the interview guide used to focus the conversation. The interview guide was used exibly with respect to the order of questions and the depth to which some issues were explored. The guide was pilot tested with a small sample of psychosocial providers (n = 5) prior to nalization. Analyses were completed using data analysis package ATLAS.ti 8.0 (Scienti c Software Development GmbH, Berlin, Germany) for coding by study authors (DMM, SB) who are PhD and PharmD trained researchers with experience in qualitative methods, health services research, and D&I science. All of the transcripts were double coded. The coders familiarized themselves with the data by carefully reading the transcripts. They then deductively coded the data using the constructs of the three RE-AIM dimensions: adoption, implementation, and maintenance. Discrepancies were resolved through discussion. Interviewer and analytic biases were managed during regular analysis meetings among all authors. Two study authors (DMM, SB) engaged in regular discussion of cases throughout the data analysis phase to ensure rigor. Transcribed interviews were coded by marked text with phrases indicating content of the discussions. Data appeared to become redundant (i.e., thematic saturation) following the 23rd interview during the pre-adaptive phase and following the 11th interview during the post-adaptive phase. All authors agreed that no unique responses were emerging within the data and that saturation had been reached. As practitioners had already agreed to participate, seven more interviews were completed during the pre-adaptation phase and three more during the post-adaptation phase.

Maintaining Research Quality
The Consolidated Criteria for Reporting Qualitative Research (COREQ) framework was used to guide the reporting of ndings. 11 Additionally, criteria for credibility, transferability, and con rmability were used to ensure rigor of this study. 12,13 Strategies used to address credibility included recording interviews and transcribing them; authors frequently discussing ndings; encouraging participants to pursue their own line of thinking; and searching the data for con icting patterns. 12, 14 Con rmability was addressed by rigorous review of interview transcripts, the codes used to identify them, and drafts and revisions of the ndings. 14

Results
Practitioners were primarily female (91%), from academic medical centers (82%) or community practices (18%) in the U.S. The majority of practitioners interviewed were psychologists (57%) or social workers (32%). Table 2 presents practitioner demographics. What follows are reports of key themes present in the data organized by three dimensions of the RE-AIM framework identi ed for qualitative analysis: adoption, implementation, and maintenance. The themes from participants receiving the initial training curriculum are compared with themes from participants receiving the adapted curriculum are compared in Table 4 and discussed below with representative quotes.  Generally, practitioners perceived Bright IDEAS as highly protocolized and required additional "time" to complete all steps of the intervention. Sometimes, practitioners encountered resistance from overwhelmed patients and overlooked the broad applicability of the intervention with clients.

Adoption
Overall, practitioners found the Bright IDEAS materials to be useful with their clients and helped to organize their clinical work. For example, a social worker stated, "I usually keep the worksheets and I have them take a picture with their phone of the action plan." Similar to the pre-adaptation group, a pro le of the ideal patient was identi ed by one psychologist as: patients that have a lot of stressors that tends to be ongoing. So, that could either be a diagnosis and they've just started treatment, or that could be longstanding, strained relationships with their family or their partner. In addition to that, I would say families that don't have a lot of social support speci cally, family or social support are good candidates for Bright IDEAS.
Practitioners also described introducing Bright IDEAS to families as, "…if there's some anxiety or stress, or the parent is critical about something, I…put them on my sort of mental list of okay, this might be a good idea for Bright IDEAS." Additionally, practitioners reported increased attempts to use Bright IDEAS, noted by, "since going to the training I think whenever a family approaches me with a need, I consider using the model [Bright IDEAS]." Implementation Overall, Bright IDEAS was used across diverse clinical settings. Speci cally, a psychologist reported, "I've been able to implement Bright IDEAS in all different settings. So, I've done it inpatient, I've done it on the outpatient side and, and certainly done it in clinic as well. It's possible." The implementation of Bright IDEAS was reported as being delivered "in multiple settings with multiple patients." Similarly, a social worker recounted, "I work for a nonpro t and we actually are a community-based organization. So, we go into the patient's house…and meet with them in their environment to discuss their problems…[using] Bright IDEAS." Further, practitioners planned time with clients for Bright IDEAS, "one other patient that I actually scheduled just speci cally to do Bright IDEAS…"

Maintenance
Largely, practitioners discussed ways in which they plan to continue to use Bright IDEAS long term. For example, one social worker shared, "I just consider it [Bright IDEAS] to be another very useful tool in my toolbox to use. I plan to just keep using it for families that clearly will bene t from it." The sentiment was also expressed as, "I feel like it [Bright IDEAS] is something I'm going to always continue to use. If I notice that there are certain participants or patients I have that would really bene t from having the goals." Additionally, there were examples of planned internal training, "So, on my team there's two other people, a social worker and a counselor. The plan is to teach them Bright IDEAS."

Discussion
To the authors' knowledge this is the rst systematic qualitative evaluation of the implementation of an EBCCP-recognized intervention. This manuscript describes the application of RE-AIM to identify the adoption, implementation and maintenance of Bright IDEAS in real-world settings. Based on feedback from trainees, modi cations were made to foster a more pragmatic approach to intervention delivery. The new training model fostered expanded use and acceptance of Bright IDEAS at the individual level. Modi cations were effective in improving practitioner intention and trial of the intervention. Further, practitioners identi ed opportunities to train colleagues.
There is a clear distinction between the two training groups, indicating a positive response to the adaptations. We believe the difference between the two groups could be partially explained by the fact that behavioral interventions are more di cult to de ne and standardize because of the inherent interactivity with client characteristics, preferences, and behaviors. 15 In this case, Bright IDEAS has proven to be e cacious for greater than 20 years yet factors which may affect real world application had not been previously investigated. Adaptions speci cally addressed environmental characteristics and behaviors. For example, we advised that Bright IDEAS NOT be formally introduced to families with newly diagnosed children until at least 4 weeks later, precisely because of a "not now" response upon initial implementation. 5, 6, 8, 10 Lau and colleagues, 16 observed similar results when examining adolescents and young adults' perspectives on facilitators and barriers to utilization of psychosocial program and found that "starting something new" was a barrier to utilization.
This study revealed moderating factors that may affect adoption more broadly. This nding is not surprising as Greenhalgh et al noted that standard attributes of the intervention will not ensure adoption alone. Rather, the interaction among the intervention, intended adopters, and a particular context that determines adoption rate 17 In this study, Bright IDEAS training adaptations considered practitioner experience and local context. While there was improved adoption, there is still opportunity for progress. Chambers and Norton describe the needed t between interventions and their settings and suggest ongoing learning about optimal intervention delivery over time. 18 Gathering feedback across diverse clinical settings and contexts should be a continual process to identify factors which affect adoption and implementation.
Bright IDEAS was accepted into the National Cancer Institute sponsored Speeding Research-tested Interventions (SPRINT) program, which is designed to foster, grow, and nurture an innovation ecosystem for interventionists. 19 Strategies provided by SPRINT experts were designed to assist in positioning Bright IDEAS as a more accessible, available, and user-friendly tool. Obviously, widespread uptake did not occur however future research should design for dissemination early in the planning process, with consideration for the resources and limitations of the patient, practitioner, and system in mind. 20 Summary Taken together, ndings reveal promising results that support the ability to improve the adoption, implementation and maintenance of Bright IDEAS thereby increasing national uptake in clinical settings. At the practitioner level, varying perspectives were expressed regarding capacity to incorporate Bright IDEAS into clinical work ow. Though some practitioners found some di culties early after training with implementing Bright IDEAS in the clinical setting, training adaptions seemed to resolve some of the issues. Speci cally, the identi cation of a patient pro le, increased use across diverse clinical settings, and intentions to train colleagues, is encouraging that improvements were experienced up to 12-month post training after adaptation were made to the training.
Intervention maintenance requires continual feedback from practitioners and adaptations which consider the changing clinical setting and needs of patients. Generalizable lessons learned underscore the importance of continual stakeholder engagement and buy-in for implementation assistance to ensure long term maintenance. Longitudinal follow-up post training is imperative to ensuring the sustainability of an intervention; otherwise limited maintenance should be anticipated.

Limitations And Future Directions
The limitations of this study should be addressed in future research. First, lessons learned from the national organizations devoted exclusively to childhood and adolescent cancer research and practice, may be di cult to generalize to other healthcare domains. Another limitation is the possibility of social desirability bias. That is, some providers may have responded to questions in a manner they thought consistent with the research aims. Future research with other stakeholders, such as institutional leadership and members of patient treatment teams, also would be valuable to understanding factors that affect the dissemination and implementation process.

Conclusion
This study highlights some of the issues psychosocial providers face when disseminating a new intervention and the steps that can be taken to improve implementation. Further, the attention to the t between characteristics of an intervention and the clinical setting, available resources and knowledge of potential implementers is critical for informing the implementation process about facilitators and barriers and "work arounds" to barriers. For psychologists and social workers, interventions that require shifts in schedule, additional resources, and new knowledge, we found that a blend of implementation strategies that helps to increase compatibility with existing organizational structures is critical for their implementation.
Future pediatric oncology-based psychosocial interventions should build on the current focus of addressing adoption, implementation, and maintenance issues when trials are rst initiated and explore the possibilities of interventions that aim to continually adapt to meeting the dynamic nature of the clinical environment. Maintenance requires integration of research-tested protocols into routine clinical work ow and tailored strategies to support dynamic clinical settings across institutions. Longitudinal follow-up post training is imperative to ensuring the maintenance of an intervention; otherwise limited maintenance should be anticipated. The project was approved by the Colorado Combined Institutional Review Board (COMIRB). This study was conducted in accordance with all applicable government regulations and University of Colorado research policies and procedures. Participants provided verbal consent for interviews through IRB-approved protocols.

Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Figure 1 Bright IDEAS Pneumonic.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.