Baseline and Post-Program Survey Findings
All scholars (n=20) completed the baseline survey and 17 completed the exit survey. For the interviews, 18 scholars (90% response rate) and 10 mentors (50%; percentage based on every scholar having one mentor; total number of mentors is unknown) completed the interviews. For the inaugural year, 50% of trainees were students, 45% researchers, and 5% practitioner scholars (Table 1). Scholars overall were 83% female and 67% White, 27% Asian, and 6% Hispanic. Nine scholars were students who were at the doctorate level (50%) and from fields such as decision science and health outcomes, epidemiology, and health promotion.
Competencies Related to the D&I competencies, scholars’ ratings of their competencies all increased from baseline to posttest (Table 2). Some of the competencies with the greatest change included identifying and applying techniques for stakeholder analysis/engagement (delta, d=0.65), describing relationships between organizational dimensions and D&I research (d=.0.54), identifying a process for adapting an evidence-based intervention (d=.0.53), and using evidence to evaluate and adapt D&I strategies for specific populations, context, etc. (d=0.48) Although there was not a significant increase in the total D&I competency score (M=4.39 vs M=4.56), the scholars reported significant increases from baseline to post-program on all of the subdomains (data not shown). The subdomains that reported the largest increases from the baseline survey were practice-based considerations and D&I design and analyses.
Engagement Related to the curriculum, 94.4% reported attending the PPHEIA webinars and 77.8% reported completing the NCI Dissemination and Implementation modules (Table 3). About 33% completed their project, while 55.6% partially completed it. A large majority (94%) reported receipt of mentoring or partial mentoring in the program. About 78% of scholars engaged with a CPCRN workgroup. The program components that were ranked most valuable (defined as very or moderately valuable) were the PPHEIA training (88.9%), D&I training modules (83.3%), and webinars (kickoff webinar-88.9% and selecting implementation theories and models-88.9%). Two other components were rated moderately valuable: mentoring (72.2%) and engagement with CPCRN project or workgroup (72.2%).
Satisfaction For overall ratings of the program, 77.8% reported satisfaction with the format of the activities and ability to speak of D&I concepts (Table 4). Overwhelmingly, trainees agreed that knowledge about the CPCRN increased (88.9%). About 83% reported being very or somewhat satisfied with the mentoring received, 72.2% reported strongly agreed or agreed that their mentor helped engage them with a workgroup, and 66.6% reported that their mentor expanded their network. All scholars (100%) reported that they would use the curriculum materials in the future and would recommend the program to others.
Post-Program Interviews with Scholars
Scholars reported a variety of benefits of the program. Scholars reported that learning implementation strategies, frameworks, and methods was useful (n= 6). One scholar stated, “So it's introduction to implementation science, theory, methods, and frameworks, implementation strategies. So these are also very useful. But it's always more interesting at the – at least for me it's very interesting to hear from people actually doing projects on the ground because it helps you.” Some scholars (n=4) noted that the program’s variety of teaching methods and mediums such as using websites, PowerPoints, modules, and a variety of other tools, were very beneficial aspects of the program. Some participants (n=3) found that the curriculum helped with their work outside of the program as a teaching assistant, or through a class assignment, dissertation, or another project.
Scholars were asked to share their experiences in the CPCRN Scholars program. Several scholars (n=4) shared that due to the CPCRN program they have more knowledge about D&I research: “I think one of the things that I got the most out of it was just the educational experience, getting a better grasp and understanding of D&I research and supporting building a foundation to it. That’s probably what I got the most out of it, was just through the webinars and, in the readings, became much more, I guess, knowledgeable about D&I research. That’s probably what I benefitted or enjoyed the most out of it.” Participants shared how the program was beneficial for their careers (n=4), for their own research (n=2), and for building their network (n=2). Regarding how the program benefited one’s career, one scholar shared: “I think it was a nice addition to what my work has been as a project director. I feel like there was a fair amount of overlap, and it allowed me to really be integrated into the network, and it also allowed me to have a better understanding with D&I science and its scope and its range since that's not my area of study currently…”
We assessed the scholars’ perspectives on the workgroup opportunities in the program and their involvement in workgroups. Scholars reported a positive aspect of the workgroups was the opportunity the workgroup provided to meet others, network with people, and collaborate with other scholars on projects (n=3). For example, one scholar described how she was able to collaborate with another scholar from a different institution on a K award due to shared research interests and both their expertise complemented each other. A few scholars described workgroups as inviting and reported workgroups had an inclusive, positive culture (n=2). Some workgroups met more frequently than others. Scholars involved with workgroups who met frequently found that helpful. For example, one scholar stated: “There is a biweekly meeting that we had with a smaller health equity group just to keep the project going. So that was very helpful.” In workgroups, scholars were able to get feedback from collaborators outside of their institution (n=2) and they gained an understanding of the structure and inner workings of the workgroup and its projects (n=2). One scholar stated: “And people would seek broader opinion from the group, and that is how I had an experience getting feedback from the other collaborators who are not associated with the institution. So this is again a great learning experience, especially at a student level. So it is my first time working with so many collaborators from different universities.” One major disadvantage that participants (n=5) noted was not joining a workgroup during their time in the program, and other challenges to workgroup engagement included having trouble joining a workgroup (n=1), workgroups conflicting with their class schedule (n=1), and workgroups having infrequent meetings (n=1). For example, one scholar stated: “I tried the organizational readiness one. But I think only one of the meetings actually happened. The rest of them were canceled. And the one that happened I was in an all-day board meeting. So I couldn’t attend.”
Other valuable program components were mentioned. One positive effect of the program on the scholars' networks was that it provided access to future collaborators on papers and postdoctoral positions (n=3). One participant stated: “I have made connections with a couple of faculties to get into my PhD research work. So yes, I have made connections, and it was very helpful networking with people through my participation as a scholar in this program.” Select scholars (n=3) mentioned that the program helped them expand connections through access to the established CPCRN network or professionals. Other ways the CPCRN program impacted the scholars' networks included facilitating connections with different academic centers (n=2), the ability to make connections through the workgroups (n=2), and getting to know people outside of the scholar’s home institution (n=2). Table 5 presents a joint display of some evaluation results with the program’s quantitative ratings and exemplar quotes from different types of scholars.
Post-Program Interviews with Mentors
Mentors were asked questions about the CPCRN Scholars program, including their general impressions, challenges, purpose of the program, and the most important area of the program. Mentors shared that interactions with scholars happened during individual meetings generally consisting of discussion around the scholar’s project and training activities. Interactions were reported to be smooth and easy, with some discussions broadening beyond the CPCRN scholar training. Three mentors stated that they interacted with scholars at scheduled meetings, with one participant stating, “I think we built it into our weekly or bi-weekly meetings so she would let me know how many interviews she’s done, where she is on analysis.”
Mentors shared their thoughts about the most important aspects of the Scholars’ program. Three mentors mentioned this was networking, with one participant saying, “…the most impactful was getting people integrated into this larger network. And realizing – allowing them to realize the richness of experts across the nation who are working in D&I science that they can link into. I think that was the most impactful.” Four mentors discussed their general satisfaction with the program, with one saying “I think it has been a great program and a real benefit to the participants….” Three mentors mentioned that the program allows scholars to build relationships, with one explaining the importance of keeping scholars engaged; one commented, “I think having the program is a good opportunity to get people engaged in the work of the CPCRN. I liked that, at least it seemed like, from a worker perspective, you were trying to match scholars with these different workgroups to collaborate with other investigators. I think building those networks and relationships is really important to do. The sooner you can do them, the better. Yes, I think pretty favorably of the program.” Table 6 presents key findings related to mentors’ perceptions of the program.
This mixed methods evaluation also asked about recommendations for program improvements. Open-ended questions on the survey and an interview item were included to allow respondents to suggest programmatic changes. Recommended areas for improvements from scholars included greater opportunities for networking and mentoring, more communications around program expectations, and extending the length of the program or project completion timeline. For example, many students (n=5) shared that there was little to no interaction with other scholars. One participant specifically shared, “I had a chance to interact with them {fellow CPCRN Scholars}, but not much. Just it would have been an introductory interaction kind of thing. But I was not able to continue interacting with them.” Similarly, mentors wanted more information about the program and expectations to help them with their roles.